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Executive summary 


The project 


The Summer Active Reading Programme aimed to improve reading skills and particularly 
comprehension by raising children's engagement in, and enjoyment of, reading at the transition from 
primary school to secondary school. The programme was delivered by Booktrust, an independent 
charity that aims to change lives through engaging people with reading. 


Participating pupils were gifted four book packs and invited to attend two summer events led by 
Booktrust staff at their new secondary school. The first book pack was gifted towards the end of the 
child’s final term at primary school, the second and third packs at the summer events and the final 
pack in the first term of secondary school. Volunteers, recruited by Booktrust, gifted the book packs 
and supported activities, including one to one reading, at the summer events. 


The trial examined the impact of the programme on 205 pupils from 10 schools in the north of England 
who had been identified as unlikely to achieve Level 4a or above by the end of Key Stage 2. Pupils 
who were not likely to gain at least Level 2 were not included in the trial. 


The study was funded by the Education Endowment Foundation as one of 24 projects in a themed 
round on literacy catch-up at the primary-secondary transition. Projects funded within this round aimed 
to identify effective ways to support pupils not achieving Level 4 in English at the end of Key Stage 2. 
The project was one of four funded with a particular focus on reading for pleasure. 


1. On average, pupils who participated in the programme made slightly more progress in 
reading comprehension than similar pupils who did not. However, this finding was not 
statistically significant, meaning that it could have occurred by chance. 


2. The evaluation was the first randomised controlled trial of a book-gifting programme in 
England that included a primary outcome measure of attainment, but problems with recruiting 
schools reduced the sensitivity of the evaluation and its potential to identify impact. 


3. A positive impact on enjoyment of reading was detected for pupils not in receipt of free 
school meals. However, in terms of attainment this group improved slightly less than those 
eligible for free school meals. 


4. The programme was welcomed by schools and volunteers, who tended to view it and their 
involvement with it positively. However, engaging all schools fully and securing parental 
engagement was challenging. 


5. Improving the training and support given to volunteers and enhanced support for schools 
would ensure that the programme is implemented as intended, though it is not guaranteed 
that increasing fidelity to the programme would increase impact. 
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What impact did it have? 


On average, pupils participating in the programme made more progress than those who did not. The 
size of the difference was small, and could be envisaged as saying that pupils who participated in the 
programme made approximately two months’ additional progress than those who did not. However, 
the finding was not statistically significant, meaning that it is not possible to conclude with confidence 
that the observed effect was due to the programme rather than chance. 


Pupils eligible for free school meals and pupils ineligible for free school meals both made more 
progress than similar pupils in the comparison group. However, both effects were not statistically 
significant, meaning that they could have occurred by chance. 


The impact of the programme on pupils’ enjoyment of reading and motivation to read was also 
assessed. Overall, a positive but not statistically significant effect size was found on both measures. 
However, the programme appeared to have a different impact on the enjoyment and motivation of 
pupils eligible for free school meals compared to their peers. Amongst pupils eligible and claiming free 
school meals a negative, but not statistically significant impact was found, whereas amongst pupils not 
eligible or not claiming free school meals a positive impact was found, which was statistically 
significant. 


: +0.13 
vs. Control 182 2 month N 
Intervention vs. Contro 8 (-0.12, +0.38) +2 mont fe) aaa ££ 
Intervention vs. Control + 0.22 
1 month N 
(FSM only) . (-0.24, +0.69) PS INONNS o 
Intervention vs. Control +0.08 
121 +1 months No 
(Non-FSM only) (-0.23, +0.40) 
“For more information about evidence ratings, see Appendix VII in the main evaluation report. 
Evidence ratings are not provided for sub-group analyses, which will always be less secure than overall findings 
*“For more information about cost ratings, see Appendix VIII in the main evaluation report. 


How secure is this finding? 


Overall, the evaluation findings are judged to be of moderate security. This assessment takes into 
account a number of factors including the study’s design, size and the level of drop-out. 


This evaluation was set up as an effectiveness trial. Effectiveness trials aim to test the intervention 
under realistic conditions in a large number of schools. A randomised controlled trial design was 
employed to compare outcomes of pupils receiving the intervention to similar pupils who did not. 
Reading comprehension was assessed using the GL New Group Reading Test. 


While the original design for this current trial was based upon a large sample, the number of schools 
and pupils who took part in the study was much lower than intended. 205 pupils were randomly 
allocated between intervention and comparison groups against an initial target of 900. 23 pupils 
subsequently dropped out of the trial after randomisation. Despite the recruitment and attrition issues, 
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the intervention and control groups for analysis were balanced in terms of participant numbers, free 
school meal eligibility and gender. Given the smaller than planned size of the final sample, it is very 
important to note that the absence of statistically significant effects in the study does not mean that the 
report provides evidence that the programme does not work. Rather, the achieved sample size is not 
sufficient to draw any clear conclusions. 


The sample of schools in the study was broadly comparable with other English schools with above 
average levels of disadvantage and thus the results are somewhat generalisable beyond the 
immediate context. Despite the low recruitment, the process evaluation provides rich information about 
the programme's implementation. The study provides important insights into the feasibility of the 
programme in terms of: its potential effects on outcomes; an estimate of these potential effect sizes 
(providing valuable information on sample sizes for future study); and key facilitators and barriers to 
implementation. 


There are only a few previous trials of book-gifting programmes and, of those that exist, little evidence 
has emerged that such programmes lead to improvements in children’s reading skills. Typically, 
existing studies have examined changes in parental attitudes for programmes related to very young 
children, and have detected small effects. 


How much does it cost? 


It is estimated that the project would cost between £130 and £160 per child to deliver to a minimum of 
60 and maximum of 90 children transitioning to one secondary school. This estimate includes 
resources (£90 per pupil), activity days (£9.30), training (£13.20) and salary and overheads (£17.50). 
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1. Introduction 


1.1 Intervention 


The Booktrust Summer Active Reading programme comprised gifting four book packs by volunteers to 
children, and two events held during the summer holidays. The book packs were made up of a 
selection of books and related items such as games and pens. The first pack was gifted a few weeks 
before the child left primary school, the second and third at the summer events, and the fourth when 
they started secondary school. At the first and fourth book pack gifting meeting the volunteers 
engaged children in activities intended to encourage them to use the packs. In addition, during the first 
meeting the volunteers encouraged the children to attend the summer events. The summer events, 
led by Booktrust activity leaders, were held at the secondary school that the child was due to start at 
the end of the summer holidays. The summer events included creative and physical activities, author 
and poet visits and one to one reading with volunteers. As with the first and fourth gifting sessions, at 
the summer events volunteers engaged children in activities intended to encourage them to use the 
book packs. 


The programme was one of four reading-for-pleasure trials that, together with a group of other projects 
funded by the Education Endowment Foundation, tested the effectiveness of programmes designed to 
improve children’s literacy across the transition from primary to secondary school. 


1.2 Background evidence 


There is no prior research or evaluation evidence for the Booktrust Summer Active Reading 
intervention. However, Booktrust’s Letterbox Club programme, provided specifically for children in 
foster care, uses a similar design and resources as the Summer Active Reading programme. A 
previous pre- post-test evaluation of the Letterbox Club indicated that there is evidence that the 
children who participated made significant progress in relation to their reading accuracy and 
comprehension and also their number skills (Winter, Connolly, Bell, & Ferguson, 2011). Notably, the 
research also suggested that attainment increased if Letterbox packs were supported by a summer 
school event. However, this evaluation did not have a control group and the authors made clear that it 
was not possible to conclude that these improvements were due to the effects of the programme. A 
randomised control trial to evaluate the impact of The Letterbox Club is currently underway. 


More generally, there is a substantial amount of international research in relation to book-gifting 
programmes (for a review of the literature see Burnett, Daniels & Bailey, 2014). Despite this, there are 
few randomised controlled trial evaluations of book-gifting interventions. These trials -have had very 
limited success in identifying significant effects produced by book-gifting programmes. Examples 
include: a book-gifting programme for reception class children (5-6 years), which showed no evidence 
of effects on parents and children (N=275) in frequency of booksharing, literacy attitudes or library use 
(Connolly, O’Hare & Mitchell, 2012); and a further trial conducted in Australia of a book-gifting 
programme for children aged 4-18 months (N=552) called ‘Let’s Read’ (Goldfield, 2012) showed no 
effects, after two years, on vocabulary, communication and home literacy environment or, after four 
years, on language and literacy environment measures. Another RCT of a book-gifting programme for 
children in early years settings (N=162) called ‘Bookstart Treasure Pack’ (Demack and Stevens, 2013) 
showed no significant effects on a range of outcomes including: perceptions of parents on reading with 
their child; perceptions of parents on their child's engagement with books, stories and rhymes; use and 
membership of a public library; and child book ownership. However, the trial did find one statistically 
significant effect on father's reading with their child (although the two groups were not balanced at pre- 
test on this outcome measure). One further RCT of a book-gifting programme called Bookstart+ 
(O’Hare & Connolly, 2014) — a programme for two-year-olds and their families (N=203) gifted by 
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health visitors — showed a small but significant effect on parental attitudes towards reading (d= 0.19). 
However, there were no additional significant effects on the other measured outcomes in the study _ 
shared reading behaviour and library usage. Overall, it can be concluded that potential effects of this 
type of intervention would be small thus trials of these programmes would need to be very large to 
pick up effects. The trials reported above have been fairly small and thus may have not identified 
effects. However, until a number of very large trials or comprehensive systematic reviews are 
undertaken the evidence of effectiveness is unclear. 


Free book gifting has also occurred in tandem as part of more complex literacy programmes and 
public health interventions. For example one public health programme, ‘Reach out and Read (ROR)’, 
includes a free book gifting element along with shared reading support for low income families (Reach 
out and Read, 2013). ROR operates in US paediatric clinics where health professionals, trained in 
literacy support, provide advice during health consultations. Trained waiting room readers also provide 
reading support, and doctors ‘prescribe’ free books to children and their families. ROR has been 
investigated by numerous research studies which include a few RCTs designed to assess the 
programme’s effectiveness. For example a study of 135 Hispanic families (Golova et al., 1998) found 
that intervention parents were 10 times more likely to read to their children three days per week than 
control families (Odds Ratio = 10.1). 


In conclusion, there is significant international activity in relation to book gifting programmes which 
utilise substantial amounts of financial and human resources. It is notable, however, that while there 
now exists a significant body of research on book gifting, many of these studies have not used 
research methods capable of producing the most robust evidence of the actual effectiveness of such 
programmes in improving literacy outcomes among children and their families. Of the small number of 
robust evaluations that do exist, they have provided little evidence that these programmes lead to 
improvements in children’s reading skills. The following report aims to add to this literature by being 
the first randomised controlled trial of a book-gifting programme in England that has included a 
primary outcome measure of attainment. 


1.3 Stage of development of the intervention 


The Booktrust Summer Active Reading intervention is contextualised within three early phases of 
development as outlined by Campbell et al., (2007). 


e Phase 0: Theoretical development (Why should this intervention work?); 

e Phase 1: Logic Modelling (How does it work?); 

e Phase 2: Exploratory or pilot trial (optimising trial measures and looking at potential effects 
overall and on sub-groups). 


This would map onto the Education Endowment Foundation system as an effectiveness trial." 


The Summer Active Reading programme has not previously been delivered in schools, however 
Booktrust is a long-established UK charity that has been delivering book gifting programmes since 
1992. Initially book gifting was focused on babies and pre-school children. More recently programmes 
have been implemented for primary, secondary and looked-after children. In addition to this trial, in 
2012-2013 Booktrust gifted over five million books to 2.5 million children. 


' For further information on EEF trial types see: 
http://educationendowmentfoundation.org.uk/library/eef-approach-to-process-evaluation/. 
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Transition from primary to secondary school and the associated dip in achievement is an 
acknowledged issue in educational policy in England and elsewhere, as is the need to close the gap in 
educational attainment between disadvantaged pupils and others. The Booktrust Summer Active 
Reading programme, with its focus on improving reading comprehension through engaging children in 
reading for pleasure, has the potential to support these policy priorities. 


The rationale for the present evaluation is to test rigorously whether book-gifting programmes have a 
measurable effect on students’ reading skills and comprehension, as well as a small number of other 
secondary outcomes. 


The study was funded by the Education Endowment Foundation as one of 24 projects in a themed 
round on literacy catch-up at the primary-secondary transition. Projects funded within this round aimed 
to identify effective ways to support pupils not achieving Level 4 in English at the end of Key Stage 2. 
The project was one of four funded with a particular focus on reading for pleasure. 


1.4 Evaluation objectives 


The aim of this evaluation was to measure, through a randomised controlled trial, the impact of the 
Booktrust Summer Active Reading programme on children who were not likely to achieve Level 4 in 
reading, or likely to achieve Level 4b or 4c, by the end of Key Stage 2 (KS2). The process evaluation 
aimed to examine the implementation of the programme and capture the perceptions and experiences 
of key stakeholders. 


As set out in the trial protocol and registration, the evaluation was designed to address the following 
questions: 


e What is the impact of the programme, at post-test, and following the delivery of the 
programme, on a number of specific reading outcomes for participating children? 


e Is the programme having a differential impact on children depending on: 


o their gender? 
o their socio-economic status? 


e Does the impact of the programme vary significantly with any variations in implementation 
found? 


1.5 Project team 


Intervention team 


Booktrust appointed a project manager, David Stockwell, for the Summer Active Reading programme, 
who secured school participation and organised the programme. The project manager was supported 
by other Booktrust staff including the research manager, Laura Venning. Booktrust staff recruited 
volunteers, delivered training events for the volunteers before the start of the intervention, and led the 
summer events. 


A school organiser, usually a teacher or school librarian, was recruited from each participating 
secondary school. In most cases school organisers liaised with their feeder primary schools to set up 
the project and gain parental or carer consent. Where necessary—such as where a secondary school 
had a large number of feeder primaries with only a few eligible pupils in each—Booktrust provided 
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additional support to make the links with the primary schools. Volunteers worked with the primary and 
secondary schools to arrange the distribution of the first and fourth book packs. The school organisers 
worked with Booktrust to arrange the dates for the summer events and to provide the evaluators with 
pupil data. 


Booktrust recruited 37 volunteers spread across the three cities where the trial was located. Our 
process evaluation indicates that just over half were community volunteers and a just over a quarter 
were university students. Other volunteers included retired teachers, librarians and teaching 
assistants/learning support assistants and parents/carers. Each volunteer supported one or two 
children. Volunteers engaged pupils in the activities during the first and fourth book pack gifting and at 
the summer events, where the second and third packs were gifted, to support engagement with the 
book packs. 


Evaluation team 


The evaluation was co-directed by Dr Bronwen Maxwell (SHU) who provided overall direction and led 
the process evaluation, and by Professor Paul Connolly (QUB) whose focus was the impact 
evaluation. For the latter, the project directors were supported by Sean Demack (SHU), Dr Liam 
O’Hare (QUB) and Anna Stevens (SHU). Lucy Clague (SHU) assisted with process evaluation and 
project management. Professor Guy Merchant (SHU) acted as expert literacy advisor to the project. 


1.6 Ethical review 


The evaluation was approved by the ethics committees at Sheffield Hallam University and Queen’s 
University, Belfast. Opt-in consent was sought from parents/carers prior to the trial (see Appendix VI 
for the consent form). Participating secondary schools signed a Memorandum of Understanding that 
set out their roles and responsibilities for the trial as well as the role and responsibilities of Booktrust 
and the evaluators (Appendix V). The parental consent form covered both participation in the 
evaluation and permission for the child to attend the summer events. This was distributed to parents 
through their child's primary school. Only children who had written parental/carer consent where 
included in the randomisation process. 
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2. Methodology 


2.1 Design 


This was an individually randomised trial (see Randomisation section for more detail) with two trial 
groups (control and intervention). There were limited numbers of schools in the trial (10), therefore a 
cluster randomised trial design was not deemed appropriate. Furthermore, the intervention mostly took 
place over the summer months so children were not required to be removed from class depending on 
their intervention group status (i.e. control or intervention) and as a result there was limited potential 
for contamination effects across the control and intervention groups. 


Initially, it was hoped that there would be two intervention arms, with one group receiving the 
intervention resources only (i.e. the book packs) and the other group receiving the resources as well 
as attending the Summer Active events. However, recruitment into the study was much lower than 
anticipated (see sample size section) and, therefore, the study was reduced to a control and 
intervention group (resources plus Summer Active events) only. 


2.2 Eligibility 


Booktrust initially recruited 17 secondary schools to take part in the trial, selecting schools where the 
proportion of pupils in receipt of pupil premium was well above the national average. Seven of the 
secondary schools withdrew before the start of the trial, so ten secondary schools with a total of 48 
feeder primary schools took part in the trial. The secondary schools were in three cities in Yorkshire 
and Humberside. 


Eligible participants were drawn from those pupils in Year 6 of primary school at the start of the 
intervention in June 2013 who were making the transition to the ten secondary schools involved in the 
study. Only pupils identified by their teachers as not likely to achieve Level 4 in reading by the end of 
KS2, or who were likely to achieve a Level 4b or 4c, were eligible to participate. Pupils who were not 
likely to achieve a minimum of Level 2 were not included in the study due to their need for more 
extensive additional support. Parental or carer opt-in consent was gained prior to randomisation. 


2.3 Intervention 


In summary the intervention consisted of: 


1. Gifting of book pack 1 at the child's primary school by a volunteer. 


2. Summer event 1 led by Booktrust staff and supported by volunteers at which book pack 2 was 
gifted. 


3. Summer event 2 led by Booktrust staff and supported by volunteers at which book pack 3 was 
gifted. 


4. Gifting of book pack 4 at the child's secondary school by a volunteer. 


Volunteers attended a two-day training event to prepare them for their role in the intervention prior to 
gifting the first book pack. 
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The first intervention took place before the end of the summer term in the child's primary school. A 
volunteer gifted a child a book pack which had been personalised with the child’s name on the outside. 
It was intended that the volunteer would spend up to 30 minutes with the child to look at the contents 
of the pack together, discuss the books, the child’s environment and attitude to reading and either play 
a game or read a book together. 


The second and third interventions took place at two Summer Active events during the summer 
holidays, held at least 3 weeks after the first intervention and 3 weeks apart from each other. The 
events consisted of a 9:45am-—3:00pm day led by a Booktrust activity leader at a suitable venue— 
ideally the secondary school at which the child would be starting in September. Volunteers also 
attended the events. Up to 30 children attended the days, with one adult present for every two to three 
children. A carousel of activities lasting 30 minutes each ran through the day, with a 30-minute slot set 
aside for a volunteer and child to read together. The final 30 minutes of the day involved gifting the 
next book pack, as per the first intervention. Parents were invited to attend this last part of the day to 
participate in their child’s exploration of the new pack and to reflect on their day and experiences in 
general. 


The fourth and final intervention followed the same process as the first, except in the setting of the 
child’s new secondary school. This intervention again took take place at least three weeks after the 
third intervention. If children did not attend an intervention, they were either gifted the next book pack 
at their next intervention, or the book pack was sent directly to their homes. 


The contents of the book pack was the same for each child: each pack including two books plus 
related items such as games and writing materials. The books selected and other items were chosen 
by Booktrust based on their knowledge of engaging more reluctant readers, taking note of previous 
research they had undertaken on appropriate resources. Volunteers were given a small sum of money 
to add a small item to the packs gifted at the summer events that was specific to the interests of the 
child they were supporting. 


The control group received no volunteer intervention. Four book packs were sent directly to the homes 
of children in the control group after the post-test was completed. 


Running the intervention required the recruitment of volunteers. Volunteering roles were advertised 
through: the web and social media (the Booktrust website, Facebook page and twitter feed, www.do- 
it.org.uk—the national volunteer database—and web sites of local community volunteering hubs); 
partner organisations; promotional leaflets in community venues; and through local newspapers and 
on local radio. A promotional event was held in each of the three cities where the trial took place. A 
formal recruitment process was followed and interviews conducted to ensure suitability for the role. A 
two-day training event was held in each city for volunteers. 


The resources, activities and outcomes of the programme are summarised in the programme logic 
model in Appendix IV. This model identifies the anticipated causal paths between activity and short-, 
medium- and long-term outcomes. 


The process evaluation indicated some variation from the intended programme processes, particularly 
in relation to the lower than anticipated attendance by children at the summer events and the very low 
attendance by parents at the final session of the events. In addition, a few pupils did not receive all the 
one to one support envisaged from their designated volunteer and a few children did not receive one 
of the book packs or received packs with missing items. 
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2.4 Outcomes 


The primary outcome in the study was reading comprehension. The New Group Reading Test was 
used as a post-test reading ability measure: this uses sentence completion and passage 
comprehension items.? Appendix | contains a statistical and graphical summary for this primary 
outcome. The New Group reading test was administered online through the GL online testing 
software. Independent invigilators were sent to the schools to facilitate the testing process and a 
teacher in each school oversaw this process. The invigilators were blind to the group status of the 
individual pupils. The KS2 reading score taken from the spring census 2013 (standardised national 
school literacy assessment) acted as a pre-test measure in the analysis models. 


The secondary outcome of interest was ‘reading for pleasure’ or children enjoying and fostering a love 
of books and reading. It was chosen because it was included in the programme logic model (Appendix 
IV) as a short-term attitudinal change that would lead to medium- to long-term benefits in assessed 
literacy ability. Two sub-scales from the Progress in International Reading Literacy Study (PIRLS) 
were used as a post-test reading attitudes measure related to reading for pleasure. The two scales 
were: (1) the 'Students Like Reading’ scale and (2) the 'Students Motivated to Read’ scale. Appendix | 
provides a statistical summary of these two scales and the questionnaire items that were used to 
derive them. As with the primary outcome, standardised national school literacy assessments were 
used as a pre-test measure in the secondary outcome models. 


2.5 Sample size 


As noted earlier, the original intention was to run a three-armed RCT with two intervention groups of 
differing levels of intervention ‘intensity’ alongside a ‘business as usual’ control group. This trial design, 
set out in the original protocol, had a predicted minimum detectable effect size of between 0.2 and 0.3 
with a statistical power of 0.8. However, the original design assumed that around 900 Y6 pupils would 
be recruited into the trial prior to randomisation. The actual number was much lower (223 Y6 pupils) 
and so, to maintain reasonable statistical power, a two-armed trial design was adopted that compared 
one intervention group with a ‘business as usual’ control group. 


The smaller than expected sample size and resulting design change from a three-armed to a two- 
armed RCT resulted in an increase in the predicted minimum detectable effect size (MDES) to 0.38. 
This pre-trial MDES was estimated based on a multi-site pupil-level RCT with the following indicators: 


a significance level of p=0.05 

statistical power of 80% 

effect size variability=0.05 

at 10 sites 

site size (pupils per class) = 20 

inter class correlation (blocking variable) = 0.05 
pre/post correlation = 0.5. 


This effect size is notably higher than the small level of effects found in previous trials of book-gifting 
programmes. As such, the final trial did not have sufficient statistical power to detect effects in the 
range normally predicted. 


Following the collection and processing of data, the MDES for the primary outcome (New Group 
Reading Test) was estimated more precisely. This was done by using the actual values of the intra- 


?NGRT Information available at: http://www.gl-assessment.co.uk/products/new-group-reading-test. 
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cluster correlation coefficient (ICC) and the correlation between the pre- and post-test measures. 
Please note that two sets of calculations are provided. The first assumes clustering by secondary 
school (this was the approach we used to estimate the MDES at the design stage—see above), whilst 
the second assumes clustering by primary school. The statistical analyses (multilevel models) 
presented in this report assume clustering at the primary school level because of the higher ICC. 


The following indicators were used to calculate the MDES: 


e 205 pupils at randomisation 

e 10secondary schools, 48 primary schools 

e site size (pupils per class) at randomisation = 20.5 (secondary school), 4.3 (primary school) 
e inter class correlation (blocking variable) = 0.05 (secondary school), 0.11 (primary school) 
e pre/post correlation = 0.43. 


This produced the following updated MDES estimates: 


e assuming clustering at secondary school level: MDES = 0.41 


e assuming clustering at primary school level: MDES= 0.31. 


It is with this in mind that the findings, especially those that are found not to be statistically significant, 
need to be interpreted with caution. In particular, it would be correct to conclude that the absence of 
statistically significant effects means that the trial has not provided robust evidence that the 
programme is effective. However, it would be incorrect to conclude that this also means that the trial 
has provided evidence that the programme is ineffective. This latter interpretation would be misleading 
as it could be that the programme is effective but that the trial is simply unable, due to the small 
sample achieved, to detect that effect with confidence. 


The lower than expected sample size was due in part to the drop-out of schools during the process of 
providing pupil data, or to a failure to gain parental consent to take part in the trial. The seven 
secondary schools that withdrew at this stage gave the following reasons: 


e Two schools failed to gain data on pupil characteristics and/or teacher assessments from 
primary schools. 


e Two schools cited issues with sharing data before attempting to contact their feeder primary 
schools. 


e Three schools cited a lack of capacity to provide the necessary staff time to liaise with primary 
schools or collect parental consent. 


Some feeder primary schools were unable to respond to requests or provide the data or consent forms 
by the trial deadlines. They cited reasons such as a lack of staff capacity, staff illness or absence, 
insufficient time to plan in the work, or poor communication from the secondary school. The lack of 
engagement by primary schools was exacerbated by the request for data and gaining parental 
consent coinciding with the weeks immediately prior to KS2 SATs. 


An opt-in consent methodology was necessary due to the nature of the intervention, which required 
parents to give consent for their children to attend the summer events. The non-return of consent 
forms by parents/carers also contributed to under-recruitment, further reducing the number of pupils 
able to participate in the trial. 
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2.6 Randomisation 


Randomisation was undertaken by the evaluators. The evaluator conducting the process was blind to 
any identifying school or pupil data. The randomisation was conducted at an individual level. The 
individuals were block randomised—initially by school, and then by reading level as indicated by 
teacher-predicted KS2 results, to create pairs. A random number generator (provided by STATA v13) 
assigned a number between 0 and 1 to each member of a pair and the individual with the highest 
random number in each pair was assigned to the intervention group. In the case of uneven numbers of 
children, the last individual was assigned to intervention or control respectively according to whether 
his or her allocated random number was above or below 0.5. 


2.7 Analysis 


Multilevel analyses were conducted using the STATA v13 software that took account of how 
participants were clustered into primary schools at the beginning of the trial. Because this was a 
project around the transition of young people from Year 6 to Year 7, they were also clustered 
differently at the end of the trial in relation to the different secondary schools they entered. The main 
analysis was run twice to assess whether taking into account the clustering of young people by 
primary or secondary school made a difference. Both approaches resulted in similar findings. 
Moreover, the degree of clustering effects was more evident in relation to the young people’s 
attendance at primary school, as was expected, and thus the main multilevel analysis was based on 
taking their clustering by primary school into account. The main analyses were conducted on an ‘intent 
to treat’ basis following which an examination of ‘dose’ or fidelity (as measured by attendance at the 
Booktrust summer events) was undertaken. 


Binary dummy variables were used to identify the intervention (coded “1”) and control (coded oy 
group participants. Similarly, binary dummy variables were used for free school meal eligibility 
(“1°=FSM, “O”=non-FSM) and gender (“1”=female, “O”=male) variables. A pre-test measure of Key 
Stage 2 reading score was also used within the analyses. 


The modelling was conducted in stages: first, a main effects stage followed by an interaction stage. 
The main effects models included the intervention / control dummy variable along with the KS2 pre- 
test measure, FSM and gender dummies. The main effects model assumes that any impact of the 
intervention is consistent across different participant subgroups (for example males and females; FSM 
and non-FSM participants) and so to explore whether this was a reasonable assumption, three 
interaction terms were introduced one at a time. The interaction terms included were: 
FSM*intervention (isolating FSM participants who received the intervention); gender*intervention 
(isolating females who received the intervention); and KS2 pre-test*intervention (isolating the pre-test 
scores of intervention group participants). If an interaction term was found to be statistically significant, 
a subgroup analysis was used to explore this in more depth. For example, if a gender*intervention 


3 The FSM measure identifies young people who are eligible and claiming free school meals from 
young people who are not eligible or eligible and not claiming them. Although this measure is widely 
used as a proxy for the socio-economic status of young people, it should be noted that there is a 
known undercount of FSM claimants that is estimated at approximately 200,000 (or 2%) of all 4-15 
years-olds in England (Iniesta-Martinez and Evans, 2012). Whilst FSM is a rather simple measure of 
socio-economic status and has this problem of inaccuracy, it is readily available and until better socio- 
economic detail is collected, remains likely to be the main tool for taking socio-economics into account 
in educational research in England. 
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interaction term was found to be statistically significant, separate models would be run for the male 
and female subsamples to explore this. 


2.8 Process evaluation methodology 


The process evaluation was designed to illuminate the implementation of the Summer Active Reading 
programme and assess fidelity. Key areas explored were (a) pupil, parent/carer and school 
engagement, (b) key stakeholders perspectives of the quality of the intervention and (c) programme 
outcomes and the fidelity of delivery of volunteer training, book gifting and the summer events. Data 
was collected using the following methods: 


June 2013 Observations of training and feedback from volunteers attending 


Observation of summer events and feedback from volunteers, Summer 


suly and Auigush 2013 Active staff and parents at the events 


October November 2013 | Volunteer survey 


Late October November 


5013 Pupil focus groups (these were conducted after completion of testing) 


Observation of training events and feedback from volunteers attending the training. 


The observations, conducted by the evaluation team, focused on reviewing the quality of delivery and 
training materials, volunteer engagement, and consistency of training. Two of the three training events 
were selected to enable comparison between lead trainer's approaches. Feedback, using a short 
written questionnaire, was collected from all volunteers attending the events that were observed by the 
researchers (n = 14). 


Observation of summer events and feedback from volunteers and Summer Active Reading 
staff at the events 


Four summer events were observed by the evaluation teams. The observations were carried out in 
two schools, both the first and second events were observed in the two schools. The schools were 
selected as they were located in the areas where the training sessions had been observed and 
differed in terms of how many children would be attending. The observations and feedback forms 
focused on children's engagement, quality of delivery, volunteers’ confidence in supporting children 
and whether they were using what they had learnt in their training, the consistency of the sessions, 
whether the events met the stated aims and objectives, what worked well, and what could have 
worked better. Twenty seven volunteers and five staff provided feedback via a short questionnaire. 


Survey of volunteers 


A link to an online questionnaire was sent to all volunteers approximately two weeks after the final 
book gifting at secondary school: 26 out of the 37 volunteers completed the questionnaire (response 
rate = 70.3%). The survey focused on the process and experience of the volunteers in relation to their 
training, gifting of the book packs, the summer events, and communication and support from the 
Booktrust team. Where volunteers had already completed either a training questionnaire or a summer 
activity questionnaire, the survey was routed in such a way that volunteers would not answer the same 
question twice. 
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Pupil focus groups 


Four focus groups were conducted approximately three weeks after the final book gifting. Focus 
groups were held in four secondary schools from across the geographical area covered by the project 
and a total of 31 children participated. Questions focused on the process, their experience of the 
intervention and their perceptions of impact. 
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3. Impact evaluation 


3.1 Timeline 

March—mid May 2013 Recruitment of schools 
Pre-test data (Key Stage 2 finely graded reading point score) was 

May 2013 collected from schools to identify eligible pupils (Note: KS2 data used for 

y analysis made available to the research team from DfE in December 

2013) 

May—18th June 2013 Collection of parental consent 

21st June 2013 Randomisation of pupils to intervention and control groups 

July 2013 First set of books gifted to intervention group by volunteers 


End July to mid-August 


5013 First Summer Active Reading events in secondary school settings 


Mid to end August 2013 | Second Summer Active Reading events in secondary school settings 


September 2013 Final set of books gifted to children 
Mid-October 2013 Post-test data (standardised age score from the NGRTB test) collected 
Mid-November 2013 Four pupils absent on the original test date are tested 


3.2 Participants 


Recruitment of schools was undertaken by Bookirust. Department of Education school performance 
data was used to draw up a shortlist and priority order of secondary schools to approach in three cities 
in Yorkshire and Humberside using the following weighted measures: 


Percentage of children making expected progress in English. 
Percentage of children eligible for Free School Meals. 
Percentage of children who are low attainers. 


Pon = 


Percentage of children who do not have English their first language. 


Forty secondary schools were initially invited to express an interest in participating in the project via an 
email, which was followed up by telephone calls and visits if necessary. Initially 17 of these schools 
agreed to participate. However, as some of these began to drop out Booktrust tried to boost school 
participation by publicising the trial via networks such as school librarians and Booktrust’s 
communications channels. Subsequent to this a further ten schools approached Booktrust requesting 
to participate. Of these one had the time and resources required to join the project at short notice and 
was included in the final total of ten schools who took part the trial. A Memorandum of Understanding 
(Appendix V) was signed by all participating schools. 
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Potential eligible pupils were identified by the secondary schools based on primary teachers’ 
assessments of predicted reading point score at KS2. Parental/carer consent was requested for all 
pupils who were not predicted to achieve Level 4, or who were likely to achieve a Level 4b or 4c and 
were likely to gain at least a Level 2. All pupils for whom parental/carer consent was secured were 
entered into the trial. An opt-in rate is not available as schools only provided the evaluators with data 
on the pupils where consent had been secured. Parent/carer consent documentation is appended 
(Appendix VI). 


All Year 6 pupils from the feeder primary schools were assessed for eligibility by the ten participating 
secondary schools. Only data on the 223 pupils that schools identified as eligible and for whom 
schools had collected parental/carer consent were passed to the evaluators. As the flowchart on the 
next page illustrates, of these 223 pupils, 18 were excluded prior to randomisation—five of the 
excluded pupils did not meet the inclusion criteria; for the remaining 13 the evaluators did not receive 
the parent/consent form. Of the 205 pupils who were randomised, 101 were allocated to the 
intervention group and 104 to the control group. All pupils were treated as intended. Of the 101 pupils 
allocated to the intervention group, seven were lost to follow-up and we were unable to match one to 
the NPD, so a total of 93 pupils from the intervention group were entered into the analysis. Of the 104 
pupils allocated to the control group, 14 were lost to follow-up and we were unable to match one to the 
NPD, so a total of 89 pupils from the control group were entered into the analysis. 


The number of pupils entered into the analysis for the secondary outcomes was lower than for the 
primary outcome of reading comprehension (primary outcomes n=183; secondary outcome ‘students 
like reading’, n=146; secondary outcome ‘students motivated to read', n=157). This was due to 
missing responses relating to the questionnaire components used to construct the scales for the 
secondary outcomes. See Appendix | for more details. 
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Assessed for eligibility (n=223) 


Excluded (n=18) 


Not meeting 
inclusion criteria 
(n=5) 


Declined to 
Randomised (n=205) participate (n=13) 


Other reasons (n=0) 


Allocated to intervention (n=101) Allocated to control (n=104) 


Received intervention (n=101) Received intervention (n=0) 


Did not receive intervention (n=0) Did not receive intervention (n=104) 


Lost to follow-up (n=7) Lost to follow-up (n=14) 
Discontinued intervention (n=0) eile Discontinued intervention (n=0) 


Unable to match to NPD (n=1) Unable to match to NPD (n=1) 


Analysed (n=93) Analysed (n=89) 


Excluded from primary analysis (n=8) pas Excluded from primary analysis 
(n=15) 
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3.3 School Characteristics 


Table 3.1 below shows (at school level) the average percentage of pupils with SEN statements, the 
average percentage of pupils with English not as a first language, and the average percentage of 
pupils eligible for FSM for the sample of schools in this study compared to the national average. The 
percentage of children with a statement of SEN is fairly similar, whilst the percentage of pupils with 
English not as a first language and those eligible for FSM is notably higher in the sample compared 
with the national average. 


Table 3.1 School characteristics of sample compared with national data 


Percentage of pupils with SEN statements or on 77% 5 6% 
School Action Plus 

Bedell of pupils with English not as a first 13.6% 33.6% 
anguage 

Percentage of pupils eligible for free school meals | 16.3% 32.1% 


With regard to Ofsted ratings, two of the schools fall in the ‘outstanding’ category, one school has a 
rating of ‘good’, five are rated as ‘satisfactory’, and one school has a rating of ‘inadequate’. At the time 
of reporting there was no published Ofsted report for one of the schools which was established in 
2011/12 (Table 3.2). 


Table 3.2 Ofsted ratings for schools 


Outstanding 2 
Good 1 
Satisfactory 5 
Inadequate 1 


All schools in the sample are in an urban setting. Four of the schools are sponsor-led academies, 
three are foundation schools, two are community schools, and one is a voluntary aided school (Table 
3.3). 


Table 3.3 School type 


Foundation School (urban) 3 


Voluntary Aided School (urban) | 1 


Community school (urban) 2 


Academy sponsor led (urban) 4 
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Table 3.4 shows that the majority of the schools are in the top two highest quintiles in terms of 
eligibility for FSM. 


Table 3.4 Pupils eligible for FSM 


Highest quintile 4 
Second highest quintile 4 
Middle quintile 2 


Seven of the schools are in the lower attainment quartile whilst two schools are in the highest quintile 
and one school is in the middle quintile (Table 3.5). 


Table 3.5 School attainment 


Second lowest quintile 7 
Middle quintile 1 
Highest quintile 2 


3.4 Pupil characteristics 


Whilst initial recruitment to the programme was much lower than was expected, amongst those who 
were recruited to the trial and randomised, the retention rate for the intervention (93%) and control 
(86%) group participants was good (see flow diagram above). 


Table 3.6 Number of participants and schools involved in the Booktrust Summer Active 
Reading Transition RCT evaluation 


Intervention Control Intervention Control 
Participants 100 103 93 89 
Primary Schools 42 45 42 41 
Secondary Schools 10 10 10 10 


In terms of FSM, gender and the pre-test measure, randomisation resulted in intervention and control 
group samples that were comparable at baseline. The samples remained statistically comparable at 
the post-test stage suggesting that the small attrition did not introduce bias in terms of FSM, gender 
and the KS2 pre-test measure (Tables 3.7 -3.9). 
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and not registered / 
claiming 


Intervention Control Intervention Control 
Eligible and registered / | 36 (36%) 33 (382%) 33 (36%) 28 (32%) 
claiming 
Not eligible or eligible 64 (64%) 70 (68%) 60 (65%) 61 (69%) 


Total 


100 (100%) 


103 (100%) 


93 (100%) 


89 (100%) 


Value (d.f.) 


Value (d.f.) 


Pearson chi-square test 


0.36 (1) 


0.55 


0.33 (1) 


0.57 


Table 3.8 Gender statistical snapshots at baseline and outcome 


Intervention Control Intervention Control 
Female 48 (48%) 49 (48%) 45 (48%) 44 (49%) 
Male 52 (52%) 54 (52%) 48 (52%) 45 (51%) 
Total 100 (100%) 103 (100%) 93 (100%) 89 (100%) 
Value (d.f.) p= Value (d.f.) p= 
Pearson chi-square | 0.00 (1) 0.95 0.02 (1) 0.89 
test 
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Table 3.9 KS2 Pre-test (Reading) statistical snapshots at baseline and outcome 


Intervention Control Intervention Control 
Mean (sd) 22.8 (5.14) 23.0 (5.13) 22.9 (5.24) 22.9 (5.25) 
Median (IQR) 24.0 (6.6) 24.4 (6.8) 24.4 (7.0) 24.4 (7.3) 
n 100 103 93 89 

mean difference p mean difference  p 

(d.f.) (d.f.) 
t-test -0.17 (201) 0.81 +0.03 (180) 0.97 
Effect Size (Hedges | -0.03 +0.01 
g) 
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3.5 Outcomes and analysis* 


Table 3.10 Primary outcome: reading comprehension” 
(n=182 participants in 48 primary schools) 


Impact evaluation 


Pre-test Gender FSM 
interaction Interaction Interaction 
coef. s.e. coef. s.e. coef. s.e. coef. s.e. 

Group (Intervention) 1.28 1.257 1.29 1.247 1.98 1.787 0.85 1.552 
Pre-test (KS2 Reading) 0.80 0.129 0.61 0.178 0.79 0.129 0.80 0.129 
Interaction (Pre-test*Intervention) - - 0.36 0.244 - - - - 
Gender (female) 0.82 1.322 0.92 1.315 1.54 1.864 0.85 1.323 
Interaction (gender*Intervention) - - - - -1.43 2.598 — - 
FSM (eligible & claiming) -1.11 1.406 -0.86 1.406 -1.20 1.412 -1.82 2.031 
Interaction (FSM*Intervention) - - - - - - 1.32 2.762 
Constant 82.32 1.282 82.16 1.283 81.98 1.423 85.53 1.350 
-2 Log Likelihood -653.896 -652.823 -653.745 -653.782 
School-level variance (s.é.) 9.07 (5.376) 9.65 (5.479) 8.87 (5.346) 8.98 (5.346) 


Pupil-level variance (s.é.) 


70.18 (8.090) 


68.93 (7.944) 


70.19 (8.094) 


70.15 (7.944) 


Intra-Class Correlation (s.e.)* 


0.11 (0.064) 


0.12 (0.065) 


0.12 (0.065) 


0.12 (0.065) 


Hedges g effect size for main effects model (95% Confidence Intervals): +0.13 (-0.12, +0.38). 


“ See Appendix II for the approach to modelling and the key terms used in the tables in this section. 
Reading comprehension was assessed using the GL New Group Reading Test. 
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The only term found to be statistically significant in the models for the primary outcome was the KS2 
pre-test variable. A positive effect size for the intervention was identified but this did not reach 
statistical significance at the 5% level. 


Whilst the FSM*intervention interaction term was not found to be statistically significant for the primary 
outcome, it was significant for the both of the secondary outcomes. Therefore, for ‘completeness’, the 
models were run on the FSM and non-FSM subsamples for the primary outcome even though the lack 
of a statistically significant interaction term suggested that there was insufficient statistical evidence 
that the intervention had a different impact for FSM pupils compared with non-FSM pupils. 


Table 3.11 Primary outcome: reading comprehension 
Subsample analysis: participants identified as eligible and claiming FSM 
(n=61 participants in 32 primary schools) 


Pre-test Gender 
interaction Interaction 
coef. s.e. coef. s.e. coef. s.e. 

Group (Intervention) 2.24 2.361 2.48 2.345 4.76 3.482 
Pre-test (KS2 Reading) 0.83 0.251 0.50 0.381 0.81 0.251 
Interaction (Pre-test*Intervention) - - 0.58 0.506 - = 
Gender (female) 0.59 2.336 0.82 2.319 3.03 3.477 
Interaction (gender*Intervention) = - - - -4.50 4.705 
Constant 80.72 2.240 80.69 2.221 79.15 2.717 
-2 Log Likelihood -220.813 -220.173 -220.381 
School-level variance (s.é.) 3.69 (10.657) 4.66 (10.918) 1.47 (10.937) 
Pupil-level variance (s.é.) 78.04 (17.262) 75.46 (16.903) 79.01 (17.817) 
Intra-Class Correlation (s.e.) 0.05 (0.130) 0.06 (0.135) 0.02 (0.136) 


Hedges g effect size for main effects model (95% Confidence Intervals) for FSM subsample: 


=+0.22 (-0.24, +0.69). 
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Table 3.12 Primary outcome: reading comprehension 
Subsample analysis: participants identified as not eligible for FSM or eligible and not 
registered / claiming 


(n=121 participants in 42 primary schools) 


Pre-test Gender 
interaction Interaction 
coef. s.e. coef. s.e. coef. s.e. 

Group (Intervention) 0.83 1.583 0.75 1.577 1.28 2.174 
Pre-test (KS2 Reading) 0.79 0.151 0.66 0.206 = 0.78 0.152 
Interaction (Pre-test*Intervention) - - 0.27 0.292 - - 
Gender (female) 0.66 1.642 0.71 1.640 1.13 2.289 
Interaction (gender*Intervention) 7 - - - -0.96 3.211 
Constant 83.02 1.389 82.93 1.392 82.80 1.563 
-2 Log Likelihood -435.242 -434.818 -435.197 
School-level variance (s.é.) 4.64 (6.309) 5.29 (6.513) 4.72 (6.330) 
Pupil-level variance (s.é.) 73.81 (10.754) 72.76 (10.646) 73.70 (10.741) 
Intra-Class Correlation (s.é.) 0.06 (0.079) 0.07 (0.082) 0.06 (0.079) 


Hedges g effect size for main effects model (95% Confidence Intervals) for non-FSM subsample 
= +0.08 (-0.23, +0.40). 


A non-significant positive effect size for the Booktrust Summer Active Reading programme was found 
amongst FSM and non-FSM participants. The positive effect was larger for FSM participant compared 
with non-FSM participants but because neither were statistically significant we cannot reliably 
conclude that the program actually had a greater impact for FSM compared with non-FSM 
participants. 
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To summarise the analyses with respect to the primary outcome measure (GL New Group Reading 
Test), the Booktrust intervention was found to have a non-significant, positive impact (of effect size h = 
+0.13) overall. This impact was found to be stronger for FSM participants (h = +0.22) compared with 
non-FSM participants (h = +0.08) but this difference was not statistically significant and so we 
conclude that there is no evidence for the intervention having an impact on the primary outcome. 


The analyses of the primary outcome then explored fidelity by examining evidence of dosage having 
an influence. This analysis was conducted just with intervention group participants. Dosage was 
measured using attendance data for the two summer events led by Booktrust and was coded as 0 (n = 
43, 46%), 1 (n = 24, 26%) or 2 (n = 26, 28%). 


Table 3.13 Primary outcome: reading comprehension and dosage analysis 
(Intervention group only: n = 93 participants in 41 primary schools) 


Pre-test (KS2 Reading) 0.98 0.168  <0.001 
Dosage (attendance) -0.32 1.043 0.759 
Constant 83.74 1.327 <0.001 
-2 Log Likelihood -329.692 

School-level variance (s.é.) 11.07 (7.268) 

Pupil-level variance (s.e.) 60.84 (10.321) 
Intra-Class Correlation (s.e.)* 0.15 (0.094) 


The influence of attendance was found not to be statistically significant on the primary measure. 
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Pre-test Gender FSM 
interaction Interaction Interaction 
coef. s.e. coef. s.e. coef. s.e. coef. s.e. 

Group (Intervention) 1.30 0.794 1.28 0.793 0.66 1.118 2.36 0.982 
Pre-test (KS2 Reading) 0.04 0.076 -0.02 0.113 0.05 0.077 0.02 0.076 
Interaction (Pre-test* Intervention) - - 0.10 0.152 - - - - 
Gender (female) 1.65 0.812 1.70 0.815 0.94 1.178 1.54 0.806 
Interaction (gender*Intervention) - - - - 1.31 1.611 - - 
FSM (eligible & claiming) -0.39 0.846 -0.30 0.854 -0.33 0.849 1.23 1.24 
Interaction (FSM*Intervention) - - - = - = -3.01 1.689 
Constant 13.61 0.763 13.56 0.765 13.92 0.854 13.12 0.809 
-2 Log Likelihood -437.412 -437.180 -437.087 -435.865 
School-level variance (s.é.) 0.88 (1.444) 0.89 (1.466) 0.99 (1.513) 1.18 (1.562) 


Pupil-level variance (s.é.) 


22.62 (2.924) 


22.53 (2.922) 


22.42 (2.918) 


21.87 (2.861) 


Intra-Class Correlation (s.e.)* 


0.04 (0.061) 


0.04 (0.062) 


0.04 (0.064) 


0.05 (0.067) 


Hedges g effect size for main effects model (95% Confidence Intervals): +0.26 (-0.05, +0.57). 


For the ‘students like reading’ secondary outcome, the main effects models show a near-significant 
positive effect for the intervention. However, once the interaction term between FSM and the 
intervention is included, the intervention effect size becomes statistically significant. This suggests that 


° This was measured using the Progress in International Reading Study (PIRLS) 2011 ‘Students Like 


Reading’ scale. 
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the intervention was more successful with non-FSM compared with FSM participants, but to look more 
closely at this, separate models were run for FSM and non-FSM subsamples. These subsample 
models are reported in Tables 3.15 and 3.16. 


Table 3.15 Secondary outcome: ‘Students Like Reading’ scale—subsample analysis for 
participants identified as eligible and claiming FSM 
(n = 52 participants in 29 primary schools) 


Pre-test Gender 
interaction Interaction 
coef. s.e. coef. s.e. coef. s.e. 

Group (Intervention) -0.54 1.478 -0.52 1.464 = -0.71 2.084 
Pre-test (KS2 Reading) 0.06 0.158 0.26 0.252 0.06 0.159 
Interaction (Pre-test*Intervention) - - -0.33 0.324 - - 
Gender (female) 0.17 1.414 0.00 1.410 -0.02 2.138 
Interaction (gender*Intervention) - - - = 0.34 2.868 
Constant 15.06 1.341 14.93 1.333 15.16 1.608 
-2 Log Likelihood -158.249 -157.727 -158.242 
School-level variance (s.é.) 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 
Pupil-level variance (s.é.) 25.76 (5.051) 25.24 (4.951) 25.75 (5.050) 
Intra-Class Correlation (s.e.)* 0.00 (0.00) 0.00 (0.00) 0.00 (0.00) 


Hedges g effect size for main effects model (95% Confidence Intervals) for FSM subsample: 
=-0.10 (-0.66, +0.46). 


The negative FSM*intervention interaction term in the complete sample model is mirrored by the 
(negative) intervention terms for the FSM subgroup models. However, the effect size is not statistically 
significant and so the most reasonable conclusion is that the trial has provided no evidence that the 
Booktrust Summer Active Reading intervention had an effect on the 'students like reading’ outcome for 
FSM participants. 
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Table 3.16 Secondary outcome: ‘Students Like Reading’—subsample analysis: 
Participants identified as not eligible for FSM or eligible and not registered / claiming. 
(n = 94 participants in 39 primary schools) 


Pre-test Gender 
interaction Interaction 
coef. s.e. coef. s.e. coef. s.e. 

Group (Intervention) 2.23 0.951 2.15 0.945 = 1.28 1.314 
Pre-test (KS2 Reading) -0.03 0.085 = =-0.15 0.121 -0.02 0.085 
Interaction (Pre-test*Intervention) - - 0.23 0.166 - 
Gender (female) 2.58 0.967 2.72 0.959 1.57 1.380 
Interaction (gender*Intervention) - - - = 1.98 1.898 
Constant 12.71 0.799 12.64 0.791 13.16 0.903 
-2 Log Likelihood -276.521 -275.601 -275.985 
School-level variance (s.é.) 0.11 (1.490) 0.00 (0.000) 0.00 (0.000) 
Pupil-level variance (s.é.) 20.91 (3.383) 20.61 (3.007) 20.78 (3.032) 
Intra-Class Correlation (s.e.)* 0.01 (0.071) 0.00 (0.000) 0.00 (0.000) 


Hedges g effect size for main effects model (95% Confidence Intervals) for non-FSM subsample: 
=+0.46 (+0.08, +0.85). 


Amongst non-FSM participants, a statistically significant, positive effect (+0.46) is observed. A 
significant female bias was also found on the PIRLS measure. 


To summarise the analyses with respect to the ‘students like reading’ outcome; the Booktrust Summer 
Active Reading intervention was found to have a statistically significant, positive impact (of effect size 
h = +0.46) for non-FSM participants, however, no evidence of impact was observed for FSM 
participants. A female bias in the PIRLS ‘students like reading’ scale was observed for non-FSM 
participants but was not evident for FSM participants. 
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Table 3.17 Secondary outcome: ‘Students Like Reading’ scale—dosage analysis, intervention 
group only 


(n = 77 participants in 39 primary schools) 


coef. s.e p= 
Pre-test (KS2 Reading) 0.08 0.103 0.466 
Dosage (attendance) -0.02 0.662 0.974 
Constant 15.48 0.846 <0.001 
-2 Log Likelihood -231.616 
School-level variance (s.é.) 3.29 (2.853) 
Pupil-level variance (s.é.) 21.09 (4.055) 
Intra-Class Correlation (s.e.)* 0.13 (0.112) 


The analysis into dosage amongst intervention group participants found no evidence that attending the 
summer events had an impact on the PIRLS ‘students like reading’ scale (Table 3.17). 


The next analyses examine the PIRLS 'students motivated to read’ scale (Tables 3.18-3.21). 
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(n = 157 participants in 47 primary schools) 


Impact evaluation 


Pre-test Gender FSM 
interaction Interaction Interaction 
coef. s.e. coef. s.e. coef. s.e. coef. s.e. 

Group (Intervention) 0.63 0.503 0.63 0.503 0.74 0.708 1.34 0.620 
Pre-test (KS2 Reading) 0.00 0.049 0.04 0.070 0.00 0.049 -0.01 0.049 
Interaction (Pre-test*Intervention) - - -0.07 0.096 - - - - 
Gender (female) 0.85 0.519 0.82 0.521 0.97 0.737 0.80 0.513 
Interaction (gender*Intervention) - - - - -0.23 1.029 - - 
FSM (eligible & claiming) 0.37 0.549 0.34 0.550 0.36 0.550 1.48 0.772 
Interaction (FSM*Intervention) - - - = - = -2.13 1.075 
Constant 13.35 0.496 13.37 0.495 13.29 0.552 13.01 0.513 
-2 Log Likelihood -406.028 -405.789 -406.002 -404.120 
School-level variance (s.é.) 0.65 (0.795) 0.61 (0.781) 0.64 (0.795) 0.44 (0.761) 
Pupil-level variance (s.é.) 9.75 (1.263) 9.75 (1.262) 9.76 (1.266) 9.68 (1.265) 
Intra-Class Correlation (s.e.)* 0.06 (0.075) 0.06 (0.074) 0.06 (0.075) 0.04 (0.075) 


Hedges g effect size for main effects model (95% Confidence Intervals): +0.19 (-0.11, +0.49). 


Once more, for 'students motivated to read’, the intervention effect emerges to become statistically 
significant (and positive) once the FSM*intervention term is fitted. The FSM*intervention itself is 


’ This was measured using the Progress in International Reading Study (PIRLS) 2011 ‘Students 


Motivated to Read' Scale 
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negative and statistically significant. To look more closely at this, the models were re-run on FSM and 
non-FSM subsamples and these are shown in the next two tables. 


Table 3.19 Secondary outcome: 'Siudents Motivated to Read' scale subsample analysis— 
participants identified as eligible and claiming FSM 
(n = 54 participants in 31 primary schools) 


Pre-test Gender 
interaction Interaction 
coef. s.e. coef. s.e. coef. s.e. 

Group (Intervention) -0.68 0.920 -0.81 0.921 -0.56 1.338 
Pre-test (KS2 Reading) 0.06 0.096 0.15 0.142 0.06 0.097 
Interaction (Pre-test* Intervention) - - -0.17 0.192 - - 
Gender (female) 0.42 0.909 0.31 0.910 0.53 1.319 
Interaction (gender*Intervention) = - - = -0.23 1.869 
Constant 14.71 0.860 14.77 0.852 14.64 1.027 
-2 Log Likelihood -141.649 -141.262 -141.612 
School-level variance (s.é.) 1.30 (2.198) 1.05 (2.235) 1.28 (2.198) 
Pupil-level variance (s.e.) 9.90 (2.697) 9.97 (2.766) 9.91 (5.813) 
Intra-Class Correlation (s.e.)* 0.12 (0.192) 0.09 (0.200) 0.11 (0.192) 


Hedges g effect size for main effects model (95% Confidence Intervals) for FSM subsample: 


= -0.20(-0.73, +0.33). 


Once again, the negative FSM*intervention interaction term in the original model is mirrored by the 
intervention terms for the FSM subgroup models. However, as the effect size is not statistically 
significant, the most reasonable conclusion is that the trial has provided no evidence that the 
Booktrust Summer Active Reading intervention had an effect on the 'students motivated to read’ 
outcome. 
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Table 3.20 Secondary outcome: 'Siudents Motivated to Read' scale—participanis identified as 
not eligible for FSM or eligible and not registered / claiming 
(n = 103 participants in 38 primary schools) 


Pre-test Gender 
interaction Interaction 
coef. s.e. coef. s.e. coef. s.e. 

Group (Intervention) 1.36 0.606 1.39 0.608 1.45 0.832 
Pre-test (KS2 Reading) -0.05 0.055 -0.02 0.078 -0.05 0.055 
Interaction (Pre-test*Intervention) - - -0.06 0.110 - = 
Gender (female) 1.11 0.608 1.07 0.611 1.21 0.871 
Interaction (gender*Intervention) - - - - -0.20 1.219 
Constant 12.86 0.509 12.87 0.509 12.82 0.581 
-2 Log Likelihood -261.558 -261.398 -261.545 
School-level variance (s.é.) 0.00 (0.000) 0.00 (0.000) 0.00 (0.000) 
Pupil-level variance (s.é.) 9.40 (1.310) 9.37 (1.306) 9.400 (1.310) 
Intra-Class Correlation (s.e.)* 0.00 (0.000) 0.00 (0.000) 0.00 (0.000) 


Hedges g effect size for main effects model (95% Confidence Intervals) for non-FSM subsample: 


= +0.43 (40.06, +0.81). 


Amongst non-FSM participants, a statistically significant, positive effect (+0.43) is observed. 


To summarise the subgroup analyses with respect to the ‘students motivated to read' outcome; the 
Booktrust Summer Active Reading intervention was found to have a statistically significant, positive 
impact (of effect size h = +0.43) for non-FSM participants, however, no evidence of impact was 


observed for FSM participants. 
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3.21 Secondary outcome: PIRLs 'Students Motivated to Read’ scale—dosage analysis 
(n = 80, intervention group only) 


coef. s.e p= 
Pre-test (KS2 Reading) -0.06 0.068 0.416 
Dosage (attendance) -0.09 0.486 0.831 
Constant 14.57 0.508  <0.001 
-2 Log Likelihood -208.154 
School-level variance (s.é.) 0.00 (0.000) 
Pupil-level variance (s.é.) 10.65 (1.685) 
Intra-Class Correlation (s.e.)* 0.00 (0.000) 


No evidence was found that attending the summer events had an impact on the ‘motivated to read' 
outcome. 


3.6 Cost 


The project costs between £130 and £160 per child to deliver to a minimum of 60 and maximum of 90 
children transitioning to one secondary school. Costs depend significantly on the number of schools 
participating (and corresponding economies of scale) as well as Booktrust’s ability to secure in-kind 
support to offset the costs of procuring suitable books and resources. 


This estimate includes resources (£90 per pupil) activity days (£9.30), training (£13.20), and salary 
and overheads (£17.50). 


Additional costs to secondary schools are the time and expertise of staff required to work with 
Booktrust and their feeder primary schools to set up the programme and liaise with volunteers to 
arrange the book gifting, as well as the costs relating to ensuring that the school is available and open 
for two days over the summer holidays. 


The cost quoted per child of between £130 and £160 is an underestimate of the actual full economic 
cost of the programme which would also need to include estimates of the cost of volunteer time. 
These are important to include as they represent significant ‘opportunity costs’ in that they are 
resources that could be used on other programmes. However, an estimate of the total cost of this 
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programme would require a full economic cost-effectiveness analysis and was beyond the scope of 
this present study. 


There was no charge to the schools involved in this trial. Any necessary expenditure made by schools, 
such as postage of letters, was recharged to the project budget. 
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4. Process evaluation 


4.1 Implementation 


Exposure and dosage 


Full dosage for pupils engaged in the Booktrust Summer Active Reading programme comprised gifting 
book packs one and four by volunteers in the pupil’s primary and secondary school respectively, and 
associated one to one activities led by a volunteer plus attendance at two summer events where book 
packs two and three were distributed. As detailed in the fidelity section below, there was considerable 
variation from the intended process for gifting of book packs one and four and indicative evidence that 
some pupils did not receive the full set of packs or received packs with missing items. Just over half of 
children (54%) attended at least one of the Booktrust active events—28% of children attended both 
summer active events and 26% attended just one event. Just under half (46%) did not attend either 
event. Attendance was lower than hoped for, however Booktrust staff made considerable efforts to 
boost attendance, for example by collecting children from their homes. The main reasons for non- 
attendance were holidays and other family commitments—‘it didn't work into the schedule the 
parents/carers already had in place’ (school organiser). In addition, some parents and/or children did 
not get sufficient information about the events or did not have the motivation to attend. 


Pupil engagement and motivation 


Pupil engagement appears to have been higher with the initial book packs compared to the final one. 
In the later packs, non-book items were often mentioned before the books. This may be because they 
had less time to read the books in the later packs, perhaps due to the pressures of secondary school 
work or because there had been less intensive activity around the later book packs. Some pupils’ 
engagement with volunteers at the fourth book gifting was limited by their wish to return to their class 
and not miss the lesson. (Focus group and volunteer survey.) 


The summer events engaged the children in enjoyable activities related to reading; showed the 
children that reading and books are fun; gave the children an opportunity to talk about and read books 
with a volunteer on a one to one basis; and to meet an author, inspiring them to read and write. 
Children at all the observed summer events were highly engaged throughout the day, with many of 
them smiling and chatting with the volunteers about the books, as well as other aspects of their lives. 
The children appeared comfortable with the volunteers and the flexibility of the one to one sessions 
enabled all children to participate in reading to their volunteer. Booktrust staff noted stronger 
engagement at the summer events in schools in one city, where pupils were keen to write in their 
diaries compared with schools in another city where pupils preferred to draw. Booktrust staff had some 
concern about whether the events had encouraged and inspired the children to use their Summer 
Active Reading book packs in the holidays (Booktrust staff feedback, observation of summer events.) 


Engagement in different activities at the summer events appeared to be influenced by group size. For 
example, the session with a poet worked best where there was only a small group of children and they 
could receive more individual attention, whereas the outside games worked better for larger groups. 
Boys and children who enjoy sports engaged more enthusiastically in the outside games. Pupils’ 
concentration levels varied according to the time of day. Activities that were found to be less 
successful in engaging children, such as the poem with a clue to guess what was in the next book 
pack, were dropped from subsequent events. The individual presents that were put in the book packs 
by volunteers at the summer events, were received enthusiastically. 


Some of the summer event activities had been designed to include ‘hidden’ reading elements, 
however these were less effective in engaging the children in reading activity. For example, the 
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children were supposed to read the instructions for the origami water bombs but this was often done 
by the volunteers instead. (Booktrust staff feedback, observation of summer events.) 


Parental/carer engagement 


The intended programme aim of engaging parents/carers in the final activity of the day at the summer 
events (to give them ideas about how to support their child’s reading and writing) was not achieved. At 
one observed event parents were engaged in the book gifting part of the event and stayed to chat with 
staff and volunteers to find out what their child had been doing: ‘Parents witnessed their child’s 
excitement whilst pulling out all the books. It was a special time for the parents to see and reflect on 
what their children had done throughout the day, but also share the excitement of the gifting of the 
book pack’ (Booktrust summer event activity leader). However, this was not replicated across the vast 
majority of summer events. Low parent/carer engagement may, to some extent, have been due to 
parents/carers own attitudes towards schooling, The Summer Active project leader noted that, of the 
parents who did collect their children at the end of the event, only a few appeared comfortable in the 
classroom environment. Parent/carer engagement was also higher in the secondary schools that had 
encouraged them to participate in events and communicate with the school prior to their child moving 
to the school. (Booktrust staff feedback, observation of summer events.) 


School engagement 


There was variation in the enthusiasm and commitment of schools involved in the Summer Active 
Reading programme. Where school organisers were less committed to the programme more fidelity 
issues arose in relation to gifting the first and particularly the fourth book packs. For example, in a very 
few instances lack of cooperation from the school meant that the volunteer reported that they were not 
able to make arrangements to meet with their child for the fourth book-gifting. The degree of school 
engagement and commitment also appeared to affect other aspects of programme fidelity, such as 
pupil attendance at the summer schools and parental engagement, and consequently the quality of 
the experience for pupils. These issues are discussed in more detail elsewhere in this chapter. 
Furthermore the degree of engagement of the secondary school and the strength of their relationships 
with their feeder primary schools was influential in determining whether or not primary schools were 
willing to part in the programme. (Discussion with school organisers at pupil focus groups; Booktrust 
staff feedback; volunteer survey.) 


Stakeholder perspectives and quality of the intervention 


Pupils’ perceptions and experience of the Summer Active Reading programme 


Our findings from pupil focus groups indicate that the majority of children were excited when they 
heard about Summer Active Reading programme book packs and events from the volunteer at the first 
book gifting, although a few did not want to participate and thought it was going to be boring. 


There was some variation between schools in pupils’ enthusiasm for, and interest in, the book packs. 
Enthusiasm for the books in the pack was particularly high in one school where the school organiser 
noted that the pupils selected for the intervention already had confidence in reading and parental 
support. In some schools there was more interest in the non-book items. (Pupil focus groups.) 


Pupils who had attended the summer events had a good recollection of the activities that they had 
taken part in, and recalled the events as being ‘fun' and a good way to meet new pupils from the 
secondary school that they would be moving to in September. However, a few children reported that 
they had not really engaged with the books during the events. (Pupil focus groups.) 


Boys in the focus groups were generally more positive about the programme than the girls. This may 
however be at least partly because they tended to be more confident and vocal participants in the 
focus groups. 
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Volunteers’ perceptions and experience 


The survey of volunteers indicated that they generally felt that the training had provided them with 
sufficient information about the importance of reading for pleasure, their role as a volunteer and the 
one to one reading sessions, but less about the techniques and strategies for engaging individual 
children, the group activities and the actual children themselves. This led to some volunteers in one of 
the cities feeling unprepared for the Summer Active Reading events. 


Volunteers considered that the summer events were enjoyable for the children and volunteers who 
took part: they felt that the one to one focus, poet session, physical games and creative activities had 
worked well, but would have liked more prior information about the planned activities. The volunteers 
felt that events were very well organised, and that overall communication with Booktrust was good 
(Volunteer survey.) 


4.2 Fidelity 


Training events 


The aims and objectives of the training events for volunteers were clearly set out: participants 
understood what was expected of them and the content reflected the training aims. Our observations 
of training indicated a variation in the style of presentation adopted by the different Booktrust staff 
leading training at the two training events we observed, and some differences in how fully the activities 
planned for the summer events were explained. Volunteers felt most confident about what they were 
required to do at the book gifting meetings with children and the summer events when the style of 
presentation was informal and full explanations were given (Observations of training and volunteer 
survey.) 


Book gifting 


Although data returned to Booktrust indicated that the book packs were distributed as intended, in two 
of the four focus groups some of the intervention group pupils did not receive the complete set of four 
book packs or were given packs with missing items. However, since only a small proportion of pupils 
took part in focus groups we are unable to reliably estimate how many pupils did not receive four 
complete book packs. 


The intended process for gifting the first book pack was not followed consistently in all cases. The 
book packs were not always gifted by the assigned volunteers themselves and the activities 
undertaken at the first book gifting also varied. Some volunteers were not able to carry out the 
intended one to one reading and others did not play any reading games with the children or speak to 
them about the summer events. The varying level of interaction between the volunteers and the pupils 
at the first book pack gifting impacted on the level of information and encouragement given to pupils to 
attend the summer events. This may have contributed to the low attendance at the summer events. 


One secondary school, with a large number of feeder primary schools, reported that it had found it 
logistically impossible to send volunteers to gift the first book packs in the primary schools. This was 
an unexpected response as Booktrust, not schools, was responsible for managing the volunteers. It 
does, however, further emphasise the necessity for greater support for schools to ensure that they 
understand their role and responsibilities, and to encourage stronger commitment to the programme. 
(Focus groups and volunteer survey.) 


There was weaker adherence to the intended processes for gifting the fourth book pack than the first 
pack and volunteers spent less time with the children. Almost a quarter of assigned volunteers did not 
meet with their child to gift the fourth book pack. In cases where this was due to the unavailability of 
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the volunteer, Booktrust made arrangements for another volunteer to undertake the gifting. This may 
have been one of the reasons why some volunteers reported that gifting was done on a group, rather 
than an individual, basis. A few less committed schools made it difficult for volunteers to arrange to 
see pupils and one did not provide the volunteers with the gift packs. In addition, some volunteers 
reported that secondary school timetables made it difficult for them to spend more than a very short 
period of time with the pupil. Pupils in three out of the four focus groups had experienced delays in 
receiving their fourth pack. This may mean that they did not engage with the final book pack until after 
the post-intervention test, indicating that some of the pupils taking the post test may not have received 
the full programme dosage. (Volunteer survey and focus groups.) 


Summer events 


The intended itinerary for the summer events was largely implemented in all the schools. Some minor 
changes were made for schools who undertook an event later in the light of what had worked well and 
what was less successful in other schools, and to better fit the setting. For example, two activities 
were omitted and timings adjusted. Minor adaptations were made at some events due to children not 
turning up, arriving late, or having little knowledge of the Summer Active Reading programme. 


In two of the ten secondary schools, the summer events were not held on the school site due to 
building works. This limited the extent to which the programme’s aim of offering a positive experience 
in the new school to support effective transition could be met. It also limited children’s access to the 
wider school reading resources at summer events (Staff feedback and volunteer survey.) 


4.3 Outcomes 


Improvements in children’s attitudes to learning were reported by the majority of volunteers and by 
evaluators observing the summer events. Most children in the pupil focus groups reported that 
participating in the Summer Active Reading programme had increased their engagement with, and 
enjoyment of, reading. One child remarked: ‘Diary of a Wimpy Kid changed my reading, | loved it. | 
used to hate reading, now | love it. I'm reading “The Magical Detective” now’. He continued to talk in 
animated detail about the story and how much he liked it, noting:. ‘/ read some of it every day’. Some 
children shared the contents of the packs with other family members—either reading with 
parents/siblings or playing games together. There was variation across the schools in the extent to 
which children thought that participating in the Summer Active Reading programme had increased 
their enjoyment of reading. Some children—particularly those from a school where the pupils were 
already reasonably confident readers—reported that the programme had encouraged them to read a 
wider range of books, as well as those aimed at a higher ability. However, this was not the case for all 
children in the pupil focus groups, with pupils in one school reporting that the intervention had made 
no difference to their reading. 


Volunteers, Booktrust staff, school organisers and evaluator observations all indicated that the 
programme had been less successful in engaging parents or carers in their child’s reading, and that 
there was variation in the extent to which the programme had helped children feel more prepared for 
their secondary school. 


There were no reported unintended consequences or negative effects. 
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4.4 Formative findings 


Necessary conditions for the success of the Summer Active Reading programme 
Our evaluation indicates that successful implementation of the programme requires: 


e Commitment from the secondary schools involved that is underpinned by a sense of 
ownership by the school and a clear understanding of how the intervention will benefit the 
school. This is crucial to ensuring that book gifting is implemented as intended and to 
encourage parental engagement. 


e A structured opportunity for all children to engage with a volunteer on a one to one basis 
before the summer events to build their relationships before the summer events and to receive 
information about the events. 


e The recruitment and training of enthusiastic and skilled volunteers who are able to respond 
flexibly in supporting children’s engagement with reading. 


e Mechanisms to maximise the attendance at summer events such as clear and early 
communication to parents and children. While a number of strategies were adopted in this 
trial, additional strategies are needed to boost attendance. 


e Summer event activity leaders who are able to flexibly tailor activities to the setting and 
children’s interests and skill level. This trial provides a successful model for summer events 
that can be adopted in future programmes. 


e The inclusion of one to one reading with a volunteer at the summer events. 


e Strategies to engage parents/carers in supporting their children’s engagement with the book 
packs. 


The programme could be improved by replacing some of the ‘hidden’ reading activities at the summer 
events, such as reading instructions during practical activities, with activities that more obviously 
require the child to engage in reading. 


4.5 Control group activity 


As intended, the control group did not receive the book packs until after the intervention and testing 
had been concluded and did not attend the summer events. The process evaluation did not indicate 
any resentfulness by control group pupils or any rivalry between the intervention and control groups. 
However, the small scale of the process evaluation did not allow for focus groups of control group 
pupils so we are unable to make claims about their perceptions of the trial. 
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5. Conclusion 


5.1 Key conclusions 


1. On average, pupils who participated in the programme made slightly more progress in 
reading comprehension than similar pupils who did not. However, this finding was not 
statistically significant, meaning that it could have occurred by chance. 


2. The evaluation was the first randomised controlled trial of a book-gifting programme in 
England that included a primary outcome measure of attainment, but problems with recruiting 
schools reduced the sensitivity of the evaluation and its potential to identify impact. 


3. A positive impact on enjoyment of reading was detected for pupils not in receipt of free 
school meals. However, in terms of attainment this group improved slightly less than those 
eligible for free school meals. 


4. The programme was welcomed by schools and volunteers, who tended to view it and their 
involvement with it positively. However, engaging all schools fully and securing parental 
engagement was challenging. 


5. Improving the training and support given to volunteers and enhanced support for schools 
would ensure that the programme is implemented as intended, though it is not guaranteed 
that increasing fidelity to the programme would increase impact. 


The trial did not provide evidence that the Booktrust Summer Active Reading transition programme is 
effective in improving either children's reading comprehension skills or their attitudes towards reading. 
However, it would be incorrect to conclude that the lack of statistical significance means that the 
programme was ineffective. Recruitment to the trial was much lower than expected which resulted in 
changing the design from a three-armed to a two-armed RCT, and a resulting reduction in the 
minimum detectable effect size: it could be that the programme was effective but that the trial was 
simply unable—due to the small sample achieved—to detect that effect with statistical confidence. 


The subsample analyses did reveal statistically significant and positive effect sizes amongst non-FSM 
participants with respects to attitudes towards reading® but not amongst FSM participants. Within the 
non-FSM subsample, intervention group participants had statistically significantly higher scores 
compared to their non-FSM control group peers in terms of enjoyment of reading and motivation to 
read. 


8 Statistically significant positive impact amongst non-FSM participants of g = +0.46 (‘Students Like 
Reading’ - see Table 3.16) and +0.43 (‘Students Motivated to Read’ - see Table 3.20). 
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It should be noted that, amongst FSM participants, the effect sizes found with respect to attitudes 
towards reading were negative.” FSM intervention group participants had lower scores than their FSM 
control group peers relating to enjoyment of reading and motivation to read. However, these negative 
effects were relatively small compared with the positive effect sizes found amongst non-FSM 
participants and did not reach statistical significance. Therefore, while we note an observed negative 
effect amongst FSM participants with respect to attitudes towards reading, once we take the research 
design and sample size into account, we are unable to claim with confidence that this was due to the 
programme rather than being a chance finding. 


Among non-FSM participants, we can be more confident that participation in the programme resulted 
in a positive impact in terms of attitudes towards reading but this trial was not able to provide any 
evidence to explain this interesting finding as the process evaluation was not sufficient in scale or 
scope to distinguish between the experiences of FSM and non-FSM participants. Furthermore, there is 
no evidence in the literature which reports similar findings. 


Children, volunteers and schools tended to view the programme positively but there were some 
notable problems in how it was implemented. Low attendance at the summer events was a key issue 
that needs to be addressed for future cohorts, and gaining ‘buy-in’ from all participating schools is 
crucial to ensuring that the book packs are gifted as intended. It would be valuable to focus efforts on 
increasing the involvement of parents. 


The results in this trial and evidence from the literature both suggest that large trials are required to 
pick up the potentially small effects form book-gifting programmes. Thus, a further larger scale trial is 
advisable to provide a sufficiently large sample to test the effectiveness of a Booktrust Summer Active 
Reading transition programme. Whilst this trial did not find robust evidence, the indication is for a small 
positive effect on attitudes towards reading (specifically amongst non-FSM pupils). Given the low cost 
and intensity of the programme, and the issues regarding recruitment and sample size, this is a 
tentative but encouraging finding. In order to examine whether the programme genuinely has a small 
positive effect a larger scale trial would be required. This trial would enable Booktrust to make 
changes to the delivery of the programme to increase attendance at the summer events and also the 
engagement of parents and schools and to assess whether this could possibly increase the 
effectiveness of the programme. Such a trial would also provide the opportunity for a process 
evaluation that could explore possible explanations for any differential effect between FSM and non 
FSM participants. Finally, by using a longer timescale, a larger trial could explore whether positive 
impact on attitudes towards reading lead on to positive impact on attainment (rather than assuming 
that this 'change' occurs almost simultaneously as was the focus of the trial reported here). 


5.2 Limitations 


The main limitation of the impact evaluation was the under-recruitment of children to the trial. As a 
consequence this trial did not have sufficient statistical power to detect effects in the range normally 
predicted for this type of intervention. In addition, the process evaluation indicated that the method of 
selecting the sample (based on teacher assessments) may have led to the inclusion of some children 
who were already competent readers or were already reading for pleasure. 


The trial had good internal validity, with most of the children who were initially randomised being 
retained. However, the external validity and generalisability of the trial are limited as it was 
underpowered. In line with the Education Endowment Foundation's focus on raising the attainment of 
disadvantaged children, the participating schools had high proportions of FSM pupils and of pupils for 
whom English was not their first language, indicating above average levels of disadvantage. In 


Negative impact that did not reach statistical significance amongst FSM participants of g = -0.10 
(‘Students Like Reading’ - see Table 3.15) and -0.20 (‘Students Motivated to Read’ - see Table 3.19). 
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Conclusion 


addition, the sample included a mix of children in relation to gender and FSM and there is no reason to 
believe that they are unrepresentative of the wider population of pupils in schools with above average 
levels of disadvantage. 


5.3 Interpretation 


The findings of this trial are in line with the few other RCTs of book gifting programmes that have been 
reported, which have found a limited number of significant effects. These studies have tended to 
examine changes to parental attitudes and the few identified effects have been small in size. In this 
study, there were, however, promising signs regarding the improvement of children's attitudes towards 
reading, particularly for children from non-FSM households. Therefore, there is potential that this 
programme, with refinement, could improve reading attitudes which could then act as a catalyst to 
encourage children to engage with linked activities or programmes that focus more directly on 
improving reading comprehension outcomes. For example, meta-cognition and _ self-regulation 
programmes have been found to be low cost and highly effective ways to develop comprehension and 
thinking skills (EEF, 2014 - http://educationendowmentfoundation.org.uk/toolkit/meta-cognitive-and- 
self-regulation-strategies/), which could build on the improved reading attitudes provided by an 
effective book gifting programme. Adopting such an approach would depend on: (a) ascertaining the 
effectiveness of a refined version of the Summer Active Reading programme through a further fully 
powered trial; (b) studies looking at the added value of linking interventions that target both reading 
attitudes and comprehension outcomes; and (c) a full assessment of cost-effectiveness of the 
Summer Active Reading programme. 


Addressing the implementation issues relating to the attendance of children at the summer events and 
the engagement of parents in the events may offer the potential for stronger impact. There is evidence 
from a trial of Reach Out and Read (Golova et al., 1998) that book gifting programmes increase 
parental engagement in reading with their children, and the evaluation of Booktrust’s Letterbox Club 
programme (Winter, Connolly, Bell, & Ferguson, 2011) suggests that reading attainment increases 
when gift packs are supported by a summer school event. However, analysis of dosage in this trial 
showed that it had no impact on outcomes. Again, this could be due to the implementation issues and 
insufficient sample size which occurred in this current study. 


5.4 Future research and publications 


This trial has indicated the need for further testing of book gifting programmes in terms of impact on 
reading attitude, behaviours and attainment at a scale that is sufficient to identify small effect sizes. 
We recommend that this forms part of a review of a range of programmes similar in nature to the 
Booktrust Summer Active Reading programme. This trial has indicated that the effects to be measured 
are very small (g = 0.13), so a future trial would need to be designed to detect effects between 0.1 and 
0.2. However, measuring effect sizes close to 0.1 would require very large sample sizes. 


The evaluation team will be seeking to publish the trial findings in academic journals. 
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Appendices 


Appendix! Outcomes 


Table Al.1 GL New Group Reading Test statistical summary 


Mean (sd) 82.8 (9.96) 84.0 (10.01) 83.4 (9.98) 
Upper Quartile 89.0 90.0 89.3 
Median 81.0 83.0 82.0 
Lower Quartile 74.0 76.5 76.0 

Max 111.0 126.0 126.0 

Min 69.0 69.0 69.0 

n= 89 93 182 


Deriving the secondary PIRLS items—PIRLS ‘Students Like Reading’ scale 


This scale was constructed using eight questionnaire items using the guidance provided by Martin et 
al., (2012). These eight items (or components) are shown in Table Al.2 along with the responses from 
the Booktrust RCT sample. 


Table Al.2 PIRLS 'Students Like Reading’ components 


Value 0 1 2 3 


| like talking about what | read with other 32(19%) 40(24%) 48(29%) 46(28%) 166 
people 


| would be happy if someone gave mea 17(10%) 30(18%)  60(36%)  62(37%) 169 
book as a present 


| would like to have more time for reading 25 (15%) 43 (26%)  52(32%)  45(27%) 165 


| enjoy reading 14 (8%) 20(12%) 53 (32%)  81(48%) 168 
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Value 0 1 2 3 
| only read if | have to 45 (26%) 65(38%)  34(20%) 29(17%) 173 
| think reading is boring 18 (11%)  37(22%) 49(30%) 62(37%) 166 


Value 0 1 2 3 
| read for fun 24 (14%) 24(14%)  70(41%)  52(31%) 170 
| read things that | choose myself 8 (5%) 28 (17%)  78(47%) 52(31%) 166 


The ‘Students Like Reading’ scale was computed by adding together the responses to the PIRLS 
components listed above. High scores indicate positive perceptions towards reading. The reliability of 
using the eight components to create the 'Students Like Reading’ scale was assessed using the 
Cronbach alpha statistic. For the sample of participants involved in the Booktrust Summer Active 
Reading Transition RCT evaluation, Alpha was calculated to be 0.79 which was slightly lower than that 
provided for the England sample in the PIRLS documentation (0.85) but still suggesting a good degree 
of internal consistency. The table below provides a statistical summary of the PIRLS ‘Students Like 
Reading’ scale. 


Table Al.3 PIRLS 'Students Like Reading’ scale statistical summary 


Mean (sd) 14.3 (4.91) 15.6 (5.02) 15.0 (5.00) 
Upper Quartile 18.5 19.0 19.0 
Median 15.0 16.0 15.0 
Lower Quartile 11.0 13.0 12.0 

Max 24.0 24.0 24.0 

Min 1.0 0.0 0.0 

n= 69 77 146 


Deriving the secondary PIRLS items—PIRLS ‘Students Motivated to Read’ 
scale 
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This scale was constructed using six questionnaire items using the guidance provided by Martin et al., 
(2012). These items are shown in Table Al.4 along with the responses from the Booktrust Summer 
Active Reading RCT sample. 


Table Al.4 PIRLS 'Students Motivated to Read' components 


Value 0 1 2 3 

| like to read things that make me think 17 (10%) 25(14%) 71(41%) 61 (85%) 174 
It is important to be a good reader 5 (8%) 12 (7%) 37 (22%) 114(68%) 168 
My parents like it when | read 3 (2%) 13 (8%) 50 (29%)  104(61%) 170 
| learn a lot from reading 2 (1%) 17 (10%)  61(86%) 90 (53%) 170 
| need to read well for my future 5 (8%) 9 (5%) 56 (33%) 100(59%) 170 
| like it when a book helps me imagine 9 (5%) 18 (11%) | 60(36%) 81 (48%) 168 
other worlds 


The PIRLS ‘Students Motivated to Read' scale was computed by adding responses to the above 
components together. High scores indicate positive perceptions towards reading. For the sample of 
participants involved in the Booktrust Summer Active Reading Transition RCT evaluation, Alpha was 
calculated to be 0.79 for the PIRLS ‘Students Motivated to Read' scale which was slightly higher than 
that provided for the England sample in the PIRLS documentation (0.78). 


Table Al.5 provides a statistical summary of the PIRLS 'Students Motivated to Read' scale. 


Table Al.5 PIRLS 'Students Motivated to Read’ scale statistical summary 


Mean (sd) 13.9 (3.26) 14.5 (3.30) 14.2 (3.28) 
Upper Quartile 16.0 17.0 17.0 
Median 15.0 15.0 15.0 
Lower Quartile 12.5 13.0 13.0 

Max 18.0 18.0 18.0 

Min 2.0 0.0 0.0 

n= 77 80 157 
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Appendix Il Approach to modelling and terms used in 
tables 


Approach to Modelling 


The models were fitted in stages: 
Main effects model 


First, a main effects model was used that simply fits the coefficient terms for all explanatory variables. 
This assumes that no interactions between the terms are present. To explore whether this is a 
reasonable assumption, a series of models were constructed that fitted specific interactions within the 
programme under evaluation. Such models explore whether there is evidence for the programme 
having a differential (rather than consistent) impact for differing groups of participants: 


Pre-test interaction 


These models explore whether the impact of the programme under evaluation depends upon prior 
attainment (at KS2). For example, the programme might have a greater impact on higher attainers 
(indicated by a positive coefficient on the interaction term) or it might have a greater impact on lower 
attainers (a negative coefficient on the interaction term). 


Gender interaction 


These models explore whether the impact of the programme under evaluation depends upon gender. 
For example, the programme might have a greater impact for females (indicated by a positive 
coefficient on the interaction term) or it might have a greater impact on males (a negative coefficient 
on the interaction term). 


FSM interaction 


These models explore whether the impact of the programme under evaluation depends upon FSM 
status. For example, the programme might have a greater impact for FSM participants (indicated by a 
positive coefficient on the interaction term) or it might have a greater impact on non-FSM participants 
(a negative coefficient on the interaction term). 


Effect Size 


This is a standardised measure that is calculated from the model coefficient. Hedges g is the effect 
size measure used (see Appendix III for more detail on calculating this). The measure standardises so 
that units are converted into standard deviations which, unlike the raw coefficient, can be directly 
compared across many models (i.e. primary and secondary outcome measures). Higher values 
indicate greater statistical impact. 
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Key terms used in the modelling tables 


Coefficient (Coef) 


This is the estimated coefficient term for each of the explanatory variables. This term can be positive, 
zero or negative. A positive coefficient indicates a higher score on the outcome variable. For example, 
if a positive coefficient is seen for the gender variable, this tells us that when taking other explanatory 
variables included in the model into account, on average females got higher scores on the outcome 
variable. Similarly, a negative coefficient indicates a lower score on the outcome variable compared 
with their male peers. For example, if a negative coefficient is seen for the FSM variable, this tells us 
that when taking other explanatory variables included in the model into account, on average FSM 
participants got lower scores on the outcome variable compared with their non-FSM peers. A zero 
coefficient indicates that there is no differences in the outcome variable. For example, if a zero 
coefficient is seen for the pre-test variable, this tells us that scores on the outcome variable are not 
associated with pre-test scores. 


Coefficients are shown in the units of the outcome variable to which they relate which means that 
comparing coefficients of one explanatory variable across primary and secondary outcomes is 
problematic (because of the differing units). This is one reason why effect sizes are used to help 
interpret as they standardise the units and enable direct comparisons to be made. 


Standard Error (s.e.) 

This is the standard error for each coefficient term. These are used to give an indication of how 
precise the estimated coefficient term is (smaller values indicate greater precision). The standard 
error takes account of sample size and is widely used in tests of statistical significance and 
calculations of confidence intervals. 

Constant 

This is the average (mean) of the outcome variable once all of the explanatory variables have been 
included. This represents the mean score for the reference group of the model, such as non-FSM 
males who attained a mean score on their KS2 reading test. To calculate the mean for a different 
group, a coefficient term would need to be added—for example, for non-FSM females who attained a 
mean KS2 attainment score, the gender coefficient would need to be added. 

-2 Log Likelihood 

This is the total amount of variation remaining once each model has been fitted, and is conceptually 
similar to the Residual or Error Sum of Squares used within linear regression and ANOVA type 
analyses. 


School level variance 


This is the amount of school-level variation that remains once the model is fitted. A standard error term 
is included with this estimate. 


Pupil level variance 


This is the amount of pupil-level variation that remains once the model is fitted. A standard error term 
is included with this estimate. 
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Intra-Cluster Correlation (ICC) 


This is the proportion of variation that is found at the higher level of the model, i.e. the school level. It 
is usual for this proportion to be much smaller than that found at the individual, pupil, level. 
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Appendix Ill Effect Size 


An effect size is a statistical estimate of the strength of a phenomenon in standardised units. In the 
context of this research, the effect size provides an indication of the difference between the 
intervention and control groups for the three outcome measures. Although the model coefficients also 
provide an indication of this, the effect size standardises these coefficients so that they can be 
compared directly with each other and across other research studies. Without standardisation, the size 
of coefficient is dependent on the scale and units of the outcome measure and so it is not possible to 
compare these directly. 


As specified by the EEF, the effect size calculated in this report is Hedges g. This is a similar effect 
size statistic to Cohen’s d but uses a standard deviation that is pooled between the intervention and 
control groups. It also includes a slight correction to reduce the bias associated with Cohen’s d when 
dealing with small samples. The tables that follow provide some summary details on the three 
outcome measures and how Hedge's g was calculated. 


Table Alll.1 Primary outcome: GL New Group Reading Test 


Mean 82.8 84.0 83.4 
Standard deviation 9.96 10.01 9.98 
n= 89 93 182 


Pooled standard deviation* 9.99 


Referring to Table 2.11 (Section 3.5), the main effects model shows a coefficient of +1.28 for the 
intervention group with a standard error of 1.257. 


The standard error can be used to calculate 95% confidence intervals for the coefficient: 
95% Confidence Interval = Coefficient +/- (1.96 x standard error) = 1.28 +/- (1.96 x 1.257) 
Upper limit of confidence interval = 1.28 + 2.464 = 3.744 

Lower limit of confidence interval = 1.28 - 2.464 = -1.184 


The above coefficient and upper/lower confidence intervals can be converted into an effect size by 
dividing by the standard deviation. For Cohen’s d, this would be the standard deviation of the primary 
outcome for all respondents (i.e. s = 9.98) whilst for Hedges g, the pooled standard deviation is used 
(S = 9.99). 


* The pooled standard deviation is calculated using the following formula: 


y= [a Da (a D8 


ny t+ng—2 
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Converting the coefficient into the (Hedge's g) effect size: 

Hedges g = coefficient / pooled standard deviation = +1.28/9.99=+0.13 
Hedges g confidence interval upper limit = +3.74 / 9.99 = + 0.37 (rounded figure) 
Hedges g confidence interval lower limit = -1.18 / 9.99 = -0.12 


The following two tables summarise the two secondary outcomes and include the pooled standard 
deviation used to calculate the reported Hedges g. 


Table Alll.2 Secondary outcome: PIRLS ‘Students Like Reading’ 


Mean 14.3 15.6 15.0 
Standard deviation 4.91 5.02 5.00 
n= 69 rah 146 
Pooled standard deviation 4.97 


Table Alll.3 Secondary outcome: PIRLS ‘Students Motivated to Read’ 


Mean 13.9 14.5 14.2 
Standard deviation 3.26 3.30 3.28 
n= 77 80 157 
Pooled standard deviation 3.28 
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Appendices 


Appendix lV Booktrust Summer Active Reading programme logic model 


Four personalized packs containing: 


e — high-appeal books selected by a panel of 
experts 

e __literacy-related games and activities 

e stationery items to ensure that the child is 
school ready 

e additional resources. 


Child reads the books 
in the packs and 
enjoys owning them. 


Child plays and enjoys 
the literacy games with 
the volunteer. 


Child uses and enjoys 
the stationary and 
additional resources. 


Child, volunteer and 
(potentially) parent or 
carer. 


Child feels sense of importance 
and value through receiving 
personalized pack. 


Child has access to new books of 
their own. 


Child feels more prepared for 
secondary school, as they have 
been given relevant resources. 


Child reads for pleasure 
more often over the 
summer holidays. 


Improvement in child’s 
attitudes to reading. 


Child ready to learn when 
they begin secondary 
school. 


Child maintains or increases 
their level of reading ability. 


Four events to gift the packs: 


e Event 1: gifting first pack at primary 
school by volunteers. 

e Events 2 & 3: two full-day events in the 
school holidays, at the secondary school, 
run by trained professionals and with 
support from volunteers. 

e Event 4: One-to-one volunteer gifting of 
final pack at secondary school. 


One-to-one work with 
volunteers and children 
and small group work. 


Parents invited for half 
an hour of each 
summer event for pack 
gifting and information 
about the activities. 


Children, volunteers, 
parents, carers and 
trained professionals 
for summer events. 


Child finds that reading activities 
can be fun. 


Parents become more engaged in 
their child’s reading and have new 
ideas about how to support this. 


Child feels more prepared for 
secondary school, having been to 
the school in the summer. 


Improvement in child’s 
attitudes to reading. 


Child reads for pleasure 
more often over the 
summer holidays. 


Child ready to learn when 
they begin secondary 
school. 


Child maintains or increases 
their level of reading ability. 


Assumptions 


there is a dip in attainment for many children. 


The programme will be delivered to Year 6 primary pupils making the 
transition to secondary schools and cover the summer holiday period, where 


Only includes pupils identified by teacher assessment as not likely to achieve 
Level 4, or likely to achieve a Level 4b or 4c in English, by end of KS2. 
Excludes pupils assessed as unlikely to achieve Level 2 or below. 


External Factors 


Other demands on the time of teachers, parents and 


children. 


Low levels of engagement from parents and carers; low 
levels of attendance at events. 


Other reading/literacy activities taking place over the same 


period. 
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Appendix V Memorandum of understanding for 
participating schools 


Booktrust/EEF summer programme: Signing Up 


To sign up to Booktrust/EEF summer programme and receive free, high quality resources for your 
participating Year 7 starters, please fill out this form and return to Booktrust by emailing 
david.stockwell@booktrust.org.uk. In order to participate in the programme, you need to sign up 
no later than Friday 19 April 2013. 


Please read and agree the terms and conditions. 
Booktrust's role 
Booktrust will: 


1. Develop and produce high quality materials for participating children making the transition 
from Year 6 to Year 7 in summer 2013 with the aim of encouraging reading for pleasure and 
improving performance in English. 

2. Supply participating children with personalisedbookpacks at no cost to the children or the 
school. These bookpacks consist of books, literacy games and resources to encourage 
reading. Some children will also receive interventions from volunteers recruited and trained 
by Booktrust. 

3. Work with evaluators to identify the groups which children fall into and ensure that they 
receive the appropriate interventions and bookpacks at agreed dates. 

4. Manage delivery of the programme and coordination of the bookpacks and interventions. 


The independent evaluators’ role (Sheffield Hallam University and Queens University, Belfast) 
The evaluators will: 
1. Undertake an independent randomised control trial and process evaluation to assess the 
impact of the programme on young people's reading. An outline of how the trial will run is 
given on the appended information sheet. 


Your school's role 


The school agrees to nominate a senior member of staff to accept these terms and conditions and 
ensure the following. 


Your school will: 


1. Nominate a lead person to liaise with Booktrust and coordinate the programme with children 
preparing to make the transition to Year 7 in September 2013. 
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2. Identify your relevant feeder schools and assist Booktrust in establishing productive 

relationships to facilitate the identification of children through teacher assessment as 

predicted not to reach level 4 in English in their end of Key Stage 2 SATS. 

Provide appropriate support wherever possible to help Booktrust deliver the summer events. 

4. Liaise with Booktrust to facilitate the final bookgifting to children by volunteers during 
September. 

5. Provide the data and facilitate the activities that are set out as requirements for schools in 
the Independent Evaluation Information Sheet. 


Oo 


The Booktrust/EEF summer programme is administered by Booktrust and supported by the 
Education Endowment Foundation. Participation is by invitation only. 


If you have any questions, please email david.stockwell@booktrust.org.uk 


Please tick to here to indicate that you have read the independent evaluation information sheet and 
you agree to these terms and conditions: |__| 


Name: 

Email address: 
School: 
Postcode: 


Estimated number of participating children: 
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The Independent Evaluation of the Booktrust/EEF summer programme - Information Sheet 


All schools who agree to participate in the Booktrust/EEF summer programmewill be participating in a randomised 
control trial (RCT) and process evaluation that has been commissioned by the Education Endowment Foundation 
and, will be conducted by Sheffield Hallam University and Queens University, Belfast. 


The purpose of the RCT and process evaluationis to measure the impact of the Booktrust/EEF summer 
programme on a number on reading outcomes for participating learners. . This will help Booktrust decide if the 
programme should be rolled out further in the future. Summary information on the effectiveness of the project will 
be published on the Education Endowment Foundation website so other schools can decide if this would be 
useful for their pupils. 


Evaluation methods and testing 


The RCT will compare the outcomes of 900 pupils: 300 of the pupils will participate in the full Booktrust/EEF 
summer programme, 300 will receive the packs only and 300 will not participate in the programme in the first 
instance, but will receive the packs from November onwards after the evaluation is complete. 


Key stage 2 results will be used as a pre intervention test and the GL New Group Reading test will be 
administered as a post intervention test to measure reading comprehension. The post test will be supplemented 
by a measure of reading for pleasure. 


The evaluators will match pupil data supplied by schools to data held by the DfE, for example gender, ethnicity 
and key stage 2 results. This data will be used in the analysis of outcomes. No pupil names will be used in the 
analysis. 


The evaluators will collect further data through surveys of key contacts in schools and volunteers and focus 
groups with pupils. Evaluation data collected by Booktrust may also be shared with the evaluators. 


Anonymised evaluation data will be shared with the Education Endowment Foundation and may be used for 
future research. 


Selection of participants 


All participants will be at the end of year 6 at the start of the project. Eligible pupils will be those who are identified 
through teacher assessment as not likely to achieve level 4, or likely to achieve a Level 4b or 4c, in reading by the 
end of key stage 2. Pupils likely to achieve at or lower than level 2 will not be included in the study. 


When schools, parents and pupils agree to take part in the programme they will not know which of these groups 
the pupils will be in. After consent has been given, eligible pupils will be randomly assigned to either of the 
intervention groups or the control group. 


Requirements for secondary schools 


e Identify all pupils who are due to start in their school in September 2013 who meet the eligibility criteria. 
e Work with Booktrust and their primary schools to facilitate the distribution of parent/carer consent forms 
to eligible pupils. 
e Provide to the evaluators via Booktrust: 
o a list of eligible pupils' names (where consent to participate has been given), their UPN and 
primary school. 
o a separate list of KS2 teacher assessments for reading, , pupil's date of birth and gender 
matched to UPN only. 
e Make arrangements for the administration of the evaluation testin October, in advance of the October 
half term. 
o This will require all pupils in both intervention groups and the control group in the school to be 
brought together to take the test at the same time. 
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o The test will be conducted electronically; it will require computer facilities and also headphones. 
It will take between 1 hour and 1 hour 30 minutes to administer. 

o A CRB-checked researcher will come to your school to oversee the testing, but a member of 
school staff will also need to be present. 


o In advance of the day that the test is administered you will be asked to do a test run to ensure it 
works with your technology. 


Requirements for primary schools 


e Organise the distribution and collection of consent forms to parents and carers of eligible pupils. 
e As necessary facilitate the arrangement of other data collection activities e.g. surveys and focus groups. 


Ethics and data storage 


The project has been checked to ensure it meets ethical standards using the approval processes in place at 


Sheffield Hallam University and Queens University, Belfast. All data will be held securely, complying with legal 
requirements. 
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Appendix VI Parent/carer consent documents 


booktrust 


Ing € 


May 2013 


Dear parent/carer, 


Your school is taking part in a trial project for 900 children leaving Year 6 and starting Year 7 this summer. The 
project will try to help them to enjoy reading and become better readers. 


We are pleased to let you know that your child has been invited to be part of this project. Each child will receive 4 
book packs with a mix of books, pens, note pads and games in each one. Your child will be able to keep all the 
things that they get in the packs. The books have been chosen to help the children to find reading fun. 


Your child will be put into one of three groups. Some will get their first parcels in July. Others will get their parcels 
in November. Some children will also be invited to take part in two ‘fun days’ during the summer break. We will let 
you know in June which group your child will be in. 


We hope to offer the packs to lots more children next year. For this to happen, we need to make sure the project 
is helping children with their reading. 


To find this out, we will need to check your child’s reading before and after they get the parcels. This will be done 
in their school. We will also contact some parents and children to find out what you think. We will write to you or 
speak to you to do this. This work is being carried out by university staff. 


The project is being paid for by a government grant. You can read all about the project and how it works on the 
next page. Please read this information and talk to your child about the project. If you want your child to take 
part in the project fill in the slip below and return it to your child's teacher by Monday 20 May 2013. If you 
do not return the slip, your child will not take part and will not get the book packs. 


We hope that your child enjoys being part of the project. 


Yours sincerely, 
David Stockwell 


EEF Project Manager 


Booktrust | Book House | 45 East Hill | London | SW18 2QZ | 07826 937475 | 
david.stockwell@booktrust.org.uk | www.booktrust.org.uk 
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-Please return this slip to your child's teacher by 20 May 2013. 
Booktrust/EEF summer project 


| have read the information sheet and understand what is involved for my child in taking part in this trial and how information 
about my child will be used and stored. 


Igive permission for my child to take part in the Booktrusi/EEF summer project and evaluation. 


Child's: fullname! sicily an eh ered ee ln ea Ma ee 
Primary-Schoolthatithey currently attend ise. ee csstie ciel ise sees spac ld facie dee ce edanleleaedieed dee Ruan deaiteticgt het Pee dadedtee es upadyalend  aeea Seeks 


Secondary School they will start in September 2013.................. cece ccc cece eee c ee eee cee eee cece e eee eee ceased eee ee essa eseeeaeeeeeeaenenes 


SIQMCG? iu: Fisica cer eite paacnig es bgaicud Saleen eee) peda eran e ee ed et eae et i oe Parent/carer 
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Bookirust/EEF Summer Project Information Sheet 
Who is running the project? 


Booktrust will provide the packs and run the fun days. The Centre for Education and Inclusion 
Research at Sheffield Hallam University and the Centre for Effective Education at Queen's University 
Belfast will evaluate the project to see how well it helps children become better readers. The 
Education Endowment Foundation is funding the project. 


What does the evaluation involve? 


Children will be put into three groups and we will see how well each group does in their reading 
between May and October 2013. The first group will get their book packs between July and 
September. The second group will get their book packs between July and September and also attend 
two fun events over the summer. The third group will get their book packs between November 2013 
and February 2014. A computer will randomly put the children in to the groups. This is like tossing a 
coin to see who goes in which group. When you agree to take part in the project you will not know 
which group your child will be in. 


What information will you collect about my child? 


Before the project starts your child’s school will tell us about how your child is doing in reading. Later 
in the project we will ask the government to give us some more information, such as your child's key 
stage 2 results and gender. 


In October 2013 your child will be asked to take an online reading test at their secondary school. We 
will collect the results of these tests. 


We will ask some of the children to talk in small groups about the project with a researcher who will 
visit their secondary school. 


What will you do with the information you collect about my child? 


We will use the information to see how well children in the three groups do in their reading. A short 
summary of our findings will be put on the Education Endowment Foundation website. This will help 
your school and other schools see how to help children become better readers. The evaluators may 
publish the findings in an academic journal. 


No schools or children will be named in any report or publication. All information about children, 
including test results, will be held confidentially and in compliance with the Data Protection Act. 
Information about your child's reading test may be given to your child’s school so that they can use it 
to help support your child. Anonymised data (without names) will be shared between Booktrust, the 
evaluators and the Education Endowment Foundation for current and future research purposes. 


What do | do if | no longer want my child to take part in the trial? 


If during the project you no longer wish your child to take part please contact the evaluation project 
manager (details below). 


David Stockwell; Project Manager Lucy Clague: Evaluation Project Manager 
Booktrust Centre for Education and Inclusion Research 
david.stockwell@booktrust.org.uk Sheffield Hallam University 

07826 937475 |.clague@shu.ac.uk0114 225 6066 
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The final security rating for this trial is 3 @. This means that findings are moderately secure. 


The Summer Active trial was originally designed as a three-armed effectiveness trial with minimum 
detectable effect of 0.2 to 0.3 and could have achieved a maximum of 4@. However, due to low 
recruitment the trial became a two armed trial with a minimum detectable effect of 0.31 and therefore 
achieves 3@. Despite the relatively low power the trial also had low attrition of 11%, which is below 
average. The groups were balanced on observable characteristics including prior attainment and 
percentage of FSM at the baseline after attrition. 


Delivery and marking of post-testing was blind to treatment allocation. Because randomisation 
occurred within the school there is a risk of some diffusion of treatment or compensation rivalry from 
peers, although there was no evidence of this from the process evaluation. 
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Appendix VIII Cost rating 


Cost ratings are based on the approximate cost per pupil of implementing the intervention over one 
year. Cost ratings are awarded using the following criteria. 


Cost 
£ 
£E 


Education Endowment Foundation 


Description 

Very low: less than £80 per pupil per year. 
Low: up to about £170 per pupil per year. 
Moderate: up to about £700 per pupil per year. 
High: up to £1,200 per pupil per year. 

Very high: over £1,200 per pupil per year. 
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You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit www.nationalarchives.gov.uk/doc/open-government-licence/version/2 
or email: psi@nationalarchives.gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission from 
the copyright holders concerned. The views expressed in this report are the authors’ and do not 
necessarily reflect those of the Department for Education. 


This document is available for download at www.educationendowmentfoundation.org.uk. 


A Education 


Endowment 
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The Education Endowment Foundation 

9th Floor, Millbank Tower 

21-24 Millbank 

London 

SW1P 4QP 
www.educationendowmentfoundation.org.uk 


